Understanding the deduplicate error when building fat jar

I get this long list of deduplicate errors when I try to build a fat jar.

[error] java.lang.RuntimeException: deduplicate: different file contents found in the following:
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:javax/annotation/Generated.class
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/8.5.46/tomcat-annotations-api-8.5.46.jar:javax/annotation/Generated.class
[error] deduplicate: different file contents found in the following:
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:javax/annotation/ManagedBean.class
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/8.5.46/tomcat-annotations-api-8.5.46.jar:javax/annotation/ManagedBean.class
[error] deduplicate: different file contents found in the following:
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:javax/annotation/PostConstruct.class
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/8.5.46/tomcat-annotations-api-8.5.46.jar:javax/annotation/PostConstruct.class
[error] deduplicate: different file contents found in the following:
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:javax/annotation/PreDestroy.class
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/8.5.46/tomcat-annotations-api-8.5.46.jar:javax/annotation/PreDestroy.class
[error] deduplicate: different file contents found in the following:
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:javax/annotation/Priority.class
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/8.5.46/tomcat-annotations-api-8.5.46.jar:javax/annotation/Priority.class
[error] deduplicate: different file contents found in the following:
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:javax/annotation/Resource$AuthenticationType.class
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/8.5.46/tomcat-annotations-api-8.5.46.jar:javax/annotation/Resource$AuthenticationType.class
[error] deduplicate: different file contents found in the following:
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:javax/annotation/Resource.class
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/8.5.46/tomcat-annotations-api-8.5.46.jar:javax/annotation/Resource.class
[error] deduplicate: different file contents found in the following:
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:javax/annotation/Resources.class
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/8.5.46/tomcat-annotations-api-8.5.46.jar:javax/annotation/Resources.class
[error] deduplicate: different file contents found in the following:
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:javax/annotation/security/DeclareRoles.class
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/8.5.46/tomcat-annotations-api-8.5.46.jar:javax/annotation/security/DeclareRoles.class
[error] deduplicate: different file contents found in the following:
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:javax/annotation/security/DenyAll.class
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/8.5.46/tomcat-annotations-api-8.5.46.jar:javax/annotation/security/DenyAll.class
[error] deduplicate: different file contents found in the following:
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:javax/annotation/security/PermitAll.class
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/8.5.46/tomcat-annotations-api-8.5.46.jar:javax/annotation/security/PermitAll.class
[error] deduplicate: different file contents found in the following:
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:javax/annotation/security/RolesAllowed.class
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/8.5.46/tomcat-annotations-api-8.5.46.jar:javax/annotation/security/RolesAllowed.class
[error] deduplicate: different file contents found in the following:
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:javax/annotation/security/RunAs.class
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/8.5.46/tomcat-annotations-api-8.5.46.jar:javax/annotation/security/RunAs.class
[error] deduplicate: different file contents found in the following:
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:javax/annotation/sql/DataSourceDefinition.class
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/8.5.46/tomcat-annotations-api-8.5.46.jar:javax/annotation/sql/DataSourceDefinition.class
[error] deduplicate: different file contents found in the following:
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:javax/annotation/sql/DataSourceDefinitions.class
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/8.5.46/tomcat-annotations-api-8.5.46.jar:javax/annotation/sql/DataSourceDefinitions.class
[error]         at sbtassembly.Assembly$.applyStrategies(Assembly.scala:161)
[error]         at sbtassembly.Assembly$.x$1$lzycompute$1(Assembly.scala:43)
[error]         at sbtassembly.Assembly$.x$1$1(Assembly.scala:41)
[error]         at sbtassembly.Assembly$.stratMapping$lzycompute$1(Assembly.scala:41)
[error]         at sbtassembly.Assembly$.stratMapping$1(Assembly.scala:41)
[error]         at sbtassembly.Assembly$.inputs$lzycompute$1(Assembly.scala:86)
[error]         at sbtassembly.Assembly$.inputs$1(Assembly.scala:76)
[error]         at sbtassembly.Assembly$.apply(Assembly.scala:103)
[error]         at sbtassembly.Assembly$.$anonfun$assemblyTask$1(Assembly.scala:278)
[error]         at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error]         at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)
[error]         at sbt.std.Transform$$anon$4.work(Transform.scala:67)
[error]         at sbt.Execute.$anonfun$submit$2(Execute.scala:281)
[error]         at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:19)
[error]         at sbt.Execute.work(Execute.scala:290)
[error]         at sbt.Execute.$anonfun$submit$1(Execute.scala:281)
[error]         at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
[error]         at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
[error]         at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error]         at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[error]         at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error]         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[error]         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[error]         at java.base/java.lang.Thread.run(Thread.java:829)
[error] (assembly) deduplicate: different file contents found in the following:
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:javax/annotation/Generated.class
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/8.5.46/tomcat-annotations-api-8.5.46.jar:javax/annotation/Generated.class
[error] deduplicate: different file contents found in the following:
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:javax/annotation/ManagedBean.class
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/8.5.46/tomcat-annotations-api-8.5.46.jar:javax/annotation/ManagedBean.class
[error] deduplicate: different file contents found in the following:
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:javax/annotation/PostConstruct.class
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/8.5.46/tomcat-annotations-api-8.5.46.jar:javax/annotation/PostConstruct.class
[error] deduplicate: different file contents found in the following:
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:javax/annotation/PreDestroy.class
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/8.5.46/tomcat-annotations-api-8.5.46.jar:javax/annotation/PreDestroy.class
[error] deduplicate: different file contents found in the following:
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:javax/annotation/Priority.class
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/8.5.46/tomcat-annotations-api-8.5.46.jar:javax/annotation/Priority.class
[error] deduplicate: different file contents found in the following:
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:javax/annotation/Resource$AuthenticationType.class
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/8.5.46/tomcat-annotations-api-8.5.46.jar:javax/annotation/Resource$AuthenticationType.class
[error] deduplicate: different file contents found in the following:
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:javax/annotation/Resource.class
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/8.5.46/tomcat-annotations-api-8.5.46.jar:javax/annotation/Resource.class
[error] deduplicate: different file contents found in the following:
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:javax/annotation/Resources.class
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/8.5.46/tomcat-annotations-api-8.5.46.jar:javax/annotation/Resources.class
[error] deduplicate: different file contents found in the following:
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:javax/annotation/security/DeclareRoles.class
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/8.5.46/tomcat-annotations-api-8.5.46.jar:javax/annotation/security/DeclareRoles.class
[error] deduplicate: different file contents found in the following:
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:javax/annotation/security/DenyAll.class
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/8.5.46/tomcat-annotations-api-8.5.46.jar:javax/annotation/security/DenyAll.class
[error] deduplicate: different file contents found in the following:
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:javax/annotation/security/PermitAll.class
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/8.5.46/tomcat-annotations-api-8.5.46.jar:javax/annotation/security/PermitAll.class
[error] deduplicate: different file contents found in the following:
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:javax/annotation/security/RolesAllowed.class
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/8.5.46/tomcat-annotations-api-8.5.46.jar:javax/annotation/security/RolesAllowed.class
[error] deduplicate: different file contents found in the following:
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:javax/annotation/security/RunAs.class
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/8.5.46/tomcat-annotations-api-8.5.46.jar:javax/annotation/security/RunAs.class
[error] deduplicate: different file contents found in the following:
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:javax/annotation/sql/DataSourceDefinition.class
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/8.5.46/tomcat-annotations-api-8.5.46.jar:javax/annotation/sql/DataSourceDefinition.class
[error] deduplicate: different file contents found in the following:
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:javax/annotation/sql/DataSourceDefinitions.class
[error] .cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/tomcat/tomcat-annotations-api/8.5.46/tomcat-annotations-api-8.5.46.jar:javax/annotation/sql/DataSourceDefinitions.class

From what I understand this means that SBT found duplicate file names from different dependencies. I can resolve this by adding a merge strategy. But before I do that I want to understand the problem.

I ran sbt dependencyTree and found the 2 dependencies tomcat-annotation-api and javax-annotation-api come from the same library - io.jaegartracing:jaegar-client

+-io.jaegertracing:jaeger-client:1.6.0
[info]   | +-io.jaegertracing:jaeger-core:1.6.0
[info]   | | +-com.google.code.gson:gson:2.8.6
[info]   | | +-io.opentracing:opentracing-api:0.33.0
[info]   | | +-io.opentracing:opentracing-util:0.33.0
[info]   | | | +-io.opentracing:opentracing-api:0.33.0
[info]   | | | +-io.opentracing:opentracing-noop:0.33.0
[info]   | | |   +-io.opentracing:opentracing-api:0.33.0
[info]   | | |   
[info]   | | +-org.apache.thrift:libthrift:0.14.1
[info]   | | | +-javax.annotation:javax.annotation-api:1.3.2  // THIS 
[info]   | | | +-org.apache.tomcat.embed:tomcat-embed-core:8.5.46 
[info]   | | | | +-org.apache.tomcat:tomcat-annotations-api:8.5.46 // THIS

I took a look at the source code of javax-annotations-api and tomcat-annotations-api and picked one of the conflicting files ManagedBean.class. javax-version and tomcat-version.

Both have different package names. Why do we get a duplicate error when they actually different and are different in the classpath?

The complaint is specifically about javax/annotation/ManagedBean.class. Tomcat may have another class with the same simple name in a different package, but it looks like they are bundling this very class, as well.

1 Like

ah okay, didn’t realize that. So it’s only the files with exact package name (and filename) that get reported?

To my understanding, yes. The documentation says: " If multiple files share the same relative path […], the default strategy is to verify that all candidates have the same contents and error out otherwise." For compiled class files, same package + same class name ~ same relative path. The difference in content probably comes from different versions of this class (or maybe even the same source version, but somewhat different compilation context/environment).

1 Like