Need help understanding LambdaDeserializer which does very costly loadClass

taroplus · September 11, 2021, 8:15pm

Hello Scala users,
I’m using Apache Spark and seeing a heavy thread contention problem due to the classLoader.loadClass. The code is here

github.com

scala/scala/blob/2.12.x/src/library/scala/runtime/LambdaDeserializer.scala#L51


      
           * @param cache       A cache used to avoid spinning up a class for each deserialization of a given lambda. May be `null`
           * @param serialized  The lambda to deserialize. Note that this is typically created by the `readResolve`
           *                    member of the anonymous class created by `LambdaMetaFactory`.
           * @return            An instance of the functional interface
           */
          def deserializeLambda(lookup: MethodHandles.Lookup, cache: java.util.Map[String, MethodHandle],
                                targetMethodMap: java.util.Map[String, MethodHandle], serialized: SerializedLambda): AnyRef = {
            assert(targetMethodMap != null)
            def slashDot(name: String) = name.replaceAll("/", ".")
            val loader = lookup.lookupClass().getClassLoader
            val implClass = loader.loadClass(slashDot(serialized.getImplClass))
            val key = LambdaDeserialize.nameAndDescriptorKey(serialized.getImplMethodName, serialized.getImplMethodSignature)
          
            def makeCallSite: CallSite = {
              import serialized._
              def parseDescriptor(s: String) =
                MethodType.fromMethodDescriptorString(s, loader)
          
              val funcInterfaceSignature = parseDescriptor(getFunctionalInterfaceMethodSignature)
              val instantiated = parseDescriptor(getInstantiatedMethodType)
              val functionalInterfaceClass = loader.loadClass(slashDot(getFunctionalInterfaceClass))

For my use-case, the lookup class

lookup.lookupClass()

is always same class as

serialized.getImplClass

I’m wondering if we can check if it is necessary to call loadClass

    val loader = lookup.lookupClass().getClassLoader
    val className = slashDot(serialized.getImplClass)
    val implClass = if (lookup.lookupClass().getName() == className) { <--- check if those are same
      lookup.lookupClass()
    } else {
      loader.loadClass(className)
    }

obviously I know nothing about this class and its purpose, so if anybody has any idea / comment, it would be appreciated.

Thanks
-Kohki Nishio