Included jar cannot see the standard library

Has anyone ever experienced a NoClassDefFoundError when an external library tries to use the Scala standard library?

The library I’m trying to work with is Vireo, Twitter’s not-very-well-documented video processing library. It’s written in C++11 with Scala bindings.

I wrote a very simple script that transcodes a well-known meme into H-264:

package com.voatz.face.example

import com.twitter.vireo.encode.H264
import com.twitter.vireo.decode.Video
import com.twitter.vireo.mux.MP4

import com.twitter.vireo.demux.Movie
import java.nio.channels.FileChannel
import java.nio.file.Paths
import java.nio.file.StandardOpenOption


/** Transcode a video into AVI format.
  *
  * Scala implementation of the example on Twitter's original blog post
  * (https://blog.twitter.com/engineering/en_us/topics/open-source/2017/introducing-vireo.html)
  *
  */
object TranscodeToH264 extends App {
  override def main(args: Array[String]): Unit = {
    val inputPath = "/Users/dolan/Downloads/white-guy-blinking.mp4"
    val oldMovie = Movie(inputPath)
    val oldVideo = Video(oldMovie.videoTrack)
    val newVideo = H264(oldVideo, 1.5F, 1, 30F)
    val newMovie = MP4(newVideo)().byteBuffer()
    val of = FileChannel.open(Paths.get("/Users/dolan/Desktop/white-guy-blinking-h264.mp4"), StandardOpenOption.CREATE)
    of.write(newMovie)
    of.close()
  }
}

I built the jar and dropped it in my lib/ directory; but sbt run causes a class lookup failure:

[info] Running com.voatz.face.example.TranscodeToH264 
[error] (run-main-0) java.lang.NoClassDefFoundError: scala/Function1$class
[error] java.lang.NoClassDefFoundError: scala/Function1$class
[error] 	at com.twitter.vireo.common.ByteData.<init>(Data.scala:41)

The ByteData class extends the another class Data, which mixes in a trait Interval, which mixes in Function1; apparently the Scala standard library is not available to the jar even though when I mix in Function1 from my own code everything works fine.

I’m not sure how to proceed; it seems to me that if the Scala standard library is in my classpath it should be available to all linked jars. What am I missing?

What is probably happening is that Vireo was compiled against a different Scala version (e.g. 2.11.x) than the one you are using now (e.g. 2.12.x).
Anytime you get runtime errors about missing classes or methods in the scala.* namespace, this is by far the most likely cause. For more information on compatibility between Scala versions see https://docs.scala-lang.org/overviews/core/binary-compatibility-of-scala-releases.html.

While in general, mixing Scala versions can cause NoClassDefFoundError, I’m surprised about this case, because scala.Function1 has been at the same location for many versions, so I don’t see how one can get that error. Or did name mangling change?

It should be noted that if you drop a JAR in the lib folder, SBT does not resolve dependencies for that JAR. On the other hand, if the needed dependency is on the classpath, it should be fine, at least if the code uses normal classloading. Is perhaps some custom classloading taking place?

I would experiment: maybe put the standard library in lib. Or try to publish vireo to a local artifactory and load it from there. Instead of “sbt run”, try to package your app and run it outside of SBT. Check the code the custom classloaders.

Function1 is a trait, and in 2.12 the encoding of traits changed to take advantage of Java 8 default methods.

Thanks Jasper! Let me see if I got that right: before 2.12, since Java 7 interfaces cannot have method implementations, scala.Function1 has been compiled into an interface scala.Function1 and a class scala.Function1$class, and now that class has disappeared, because it is no longer needed since interfaces can contain methods as of Java 8. Makes sense.

Thanks folks! Linking with a different Scala version resolved the problem.