I’m trying to update force.com components of Salesforce2hadoop Scala application, as we identified that these outdated libraries are causing problems with Salesforce data extraction. But because the application itself is old and referencing a bunch of other libraries, I’m getting errors on ‘assembly’ run and wasn’t able to fix them. I have no Scala experience, but not looking for Scala code modifications (at least for now). > Java 9.0.4, Scala 2.11.12, Sbt 1.4.7 on Windows 10.0.18363.592 64-bit
Here are the errors:
[info] Resolving net.virtual-void#sbt-dependency-graph;0.7.4 ...
[error] Server access Error: Connection refused: connect url=http://repo.typesafe.com/typesafe/ivy-releases/net.virtual-void/sbt-dependency-graph/scala_2.10/sbt_0.13/0.7.4/ivys/ivy.xml
...
[info] Resolving com.eed3si9n#sbt-assembly;0.12.0 ...
[error] Server access Error: Connection refused: connect url=http://repo.typesafe.com/typesafe/ivy-releases/com.eed3si9n/sbt-assembly/scala_2.10/sbt_0.13/0.12.0/ivys/ivy.xml
...
[error] (*:update) sbt.ResolveException: unresolved dependency: net.virtual-void#sbt-dependency-graph;0.7.4: not found
[error] unresolved dependency: com.eed3si9n#sbt-assembly;0.12.0: not found
I tried to modify libraries’ versions to use the most recent ones, update just a few from them, but was getting similar errors. Could you please provide some hints of what could be done to be able to create a “fat” jar file?
I would try to change the resolvers and use https instead of http first.
Then, you are trying to resolve an sbt plugin for version 0.13, but said you are using sbt 1.4.7. Which one is it? (look into project/build.properties)
Replaced Java and Sbt: Java 1.8.0_281, Scala 2.11.12, Sbt 0.13.18 on Windows 10.0.18363.592 64-bit
Cleared cache and cloned Git. The only change I make is in build.properties: sbt.version = 0.13.18
Now the warning and errors I get are:
`> [warn] Found version conflict(s) in library dependencies; some are suspected to be binary incompatible:
[warn]
[warn] * org.apache.avro:avro:1.7.5 is selected over {1.7.6, 1.7.5-cdh5.0.0}
[warn] +- co.datadudes:wsdl2avro_2.11:0.2.1 (depends on 1.7.5)
[warn] +- org.kitesdk:kite-data-core:1.0.0 (depends on 1.7.5)
[warn] +- co.datadudes:salesforce2hadoop_2.11:1.2 (depends on 1.7.5)
[warn] +- org.apache.hadoop:hadoop-common:2.3.0-cdh5.0.0 (depends on 1.7.5-cdh5.0.0)
[warn] +- com.twitter:parquet-avro:1.4.1 (depends on 1.7.6)
[warn]
[warn] Run 'evicted' to see detailed eviction warnings
…
java.lang.RuntimeException: deduplicate: different file contents found in the following:
C:\Users\user.ivy2\cache\org.apache.hadoop\hadoop-yarn-common\jars\hadoop-yarn-common-2.3.0-cdh5.0.0.jar:org/apache/hadoop/yarn/factories/package-info.class
C:\Users\user.ivy2\cache\org.apache.hadoop\hadoop-yarn-api\jars\hadoop-yarn-api-2.3.0-cdh5.0.0.jar:org/apache/hadoop/yarn/factories/package-info.class
at sbtassembly.Assembly$.sbtassembly$Assembly$$applyStrategy$1(Assembly.scala:106)
at sbtassembly.Assembly$$anonfun$13.apply(Assembly.scala:123)
at sbtassembly.Assembly$$anonfun$13.apply(Assembly.scala:120)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
at sbtassembly.Assembly$.applyStrategies(Assembly.scala:125)
at sbtassembly.Assembly$.x$1$lzycompute$1(Assembly.scala:25)
at sbtassembly.Assembly$.x$1$1(Assembly.scala:23)
at sbtassembly.Assembly$.stratMapping$lzycompute$1(Assembly.scala:23)
at sbtassembly.Assembly$.stratMapping$1(Assembly.scala:23)
at sbtassembly.Assembly$.inputs$lzycompute$1(Assembly.scala:67)
at sbtassembly.Assembly$.inputs$1(Assembly.scala:57)
at sbtassembly.Assembly$.apply(Assembly.scala:83)
at sbtassembly.Assembly$$anonfun$assemblyTask$1.apply(Assembly.scala:226)
at sbtassembly.Assembly$$anonfun$assemblyTask$1.apply(Assembly.scala:223)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
at sbt.std.Transform$$anon$4.work(System.scala:63)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
at sbt.Execute.work(Execute.scala:237)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.base/java.lang.Thread.run(Thread.java:844)
[error] (*:assembly) deduplicate: different file contents found in the following:
[error] C:\Users\user.ivy2\cache\org.apache.hadoop\hadoop-yarn-common\jars\hadoop-yarn-common-2.3.0-cdh5.0.0.jar:org/apache/hadoop/yarn/factories/package-info.class
[error] C:\Users\user.ivy2\cache\org.apache.hadoop\hadoop-yarn-api\jars\hadoop-yarn-api-2.3.0-cdh5.0.0.jar:org/apache/hadoop/yarn/factories/package-info.class
[error] Total time: 154 s, completed Feb 2, 2021, 1:48:00 PM
Just find the assemblyMergeStrategy stanza in your build file and add a clause for the package-info.class files. Something like this:
assemblyMergeStrategy in assembly := {
case "package-info.class" => MergeStrategy.discard
... // other clauses
case x =>
val oldStrategy = (assemblyMergeStrategy in assembly).value
oldStrategy(x)
}
Actually, I mixed this up with module-info.class files. package-info.class files serve a different purpose. But still, try to get away with discarding them.