Scala/Spark SBT is providing error - Error compiling the sbt component 'compiler-interface

Question
1

Dear All

Can you please help me how to get out of this issue while using IntelliJ with SBT and the error pops while running the program. Compilation is fine.

Note - I am running the SPARK program first time using IntelliJ for first time. Providing this details just to have the clue.

  1. Here is my build.sbt file

sbt = 1.3.10
scalaVersion := “2.12.0”
libraryDependencies += “org.apache.spark” %% “spark-core” % “2.4.5”
libraryDependencies += “org.apache.spark” %% “spark-sql” % “2.4.5”

feel free to ask any further question. thank you

2

Your Scala version, 2.12.0, is outdated. I’d suggest trying 2.12.11.

I don’t know if your problem has anything to do with that, but it’s the first thing to try.

1 Like
3

Is there even Spark for Scala 2.12? I can only find Spark for Scala 2.11.

4

Yes Spark 2.4.0 was the first release to include support for Scala 2.12 and since 2.4.1 (or 2.4.2 I can’t remember) it is the recommended version, with Scala 2.11 being deprecated.

However, even if that was more than a year ago, major vendor like AWS EMR only provide the 2.11 release.
(probably because in Java the tradition is to always use deprecated things)

1 Like