Question about Scala 3/2.13 library compat mode and SBT cross-version conflicts

Question
1

So in honor of the new Scala 3 release I made a valiant attempt at upgrading an app yesterday. If anyone is curious to see some notes or the stream video from this experience, it’s posted here: Notion: Twitch Stream Notes 5/14/21

I made some decent progress and the SBT migration plugin worked pretty well overall, plus or minus a few confusing bits.

But I eventually got stuck on some cross-version conflicts due to mixed _2.13 and _3.x library versions among my dependencies:

[error] Modules were resolved with conflicting cross-version suffixes in ProjectRef(uri("file:/home/worace/code/contour/scala/"), "core"):
[error]    org.typelevel:cats-effect-std _2.13, _3.0.0-RC3
[error]    org.scodec:scodec-bits _2.13, _3.0.0-RC3
[error]    org.typelevel:literally _3.0.0-RC3, _2.13
[error]    co.fs2:fs2-core _2.13, _3.0.0-RC3
...etc

Basically because I’m still relying on 2.13 versions of some packages which have not upgraded yet, I end up with mixed versions due to transitive deps, even if I’ve upgraded my own immediate deps.

I think it’s something like this:

          /-- A_3
My App ---
          \-- B_2.13 (CrossVersion.for3Use2_13) -- A_2.13

Now I have both A_3 and A_2.13 in the tree, which it does not like.

And I understand why this is an error, at least in the normal SBT model – it’s trying to prevent me from accidentally mixing e.g. a 2.12 and 2.13 version of some lib which are assumed to be binary incompatible.

However my intuition of the special 2.13 ↔ 3.0 binary compat case was that SBT would somehow know to let this one slide, and treat it as just any normal version eviction case, where one eventually gets picked and I can at least try to run the application. This is just an application, not a library that I would be re-publishing. So I’m not as concerned about potentially having mixed suffixes in the tree from that perspective. Just wanted to see if I could get it to run.

There’s a somewhat related note in the Scala 3 Migration Guide which says:

it is discouraged to publish a Scala 2.13 library that depends on a Scala 3 library or vice-versa. The reason is to prevent library users from ending up with two conflicting version foo_2.13 and foo_3 of the same foo library in their classpath.

Which is exactly what I was running into. However I hoped that the fact that my case was an app rather than a library (that I would be re-publishing) meant maybe I was somehow exempted from this. But perhaps that is not the case?

Alternatively, I suppose I could do a bunch of dep override surgery by hand to exclude the conflicting suffixes (for example in the case above I could manually exclude A from B’s transitive deps). But that sounds pretty onerous and would probably make the 2.13 compatibility feature much less useful for me in practice.

It sounds like maybe I just had an incomplete understanding of how the 2.13 ↔ 3.0 lib story would be used, but curious to see what advice anyone else has on how they are handling this for standalone apps (rather than libs).

2 Likes
2

You can use a 2.13 library when you compile your app with Scala 3. But I doubt you can use a library that is compiled against library A_2.13 and link it against A_3 at runtime. I.e. you can, but I’m pretty sure it’s not guaranteed to not blow up at runtime.

4 Likes
3

I see. Is that really the case, though? I guess I thought that was the point of the 2.13 v 3.0 binary compatibility…that they would be interchangeable at runtime.

Now, obviously you could still run into trouble if you’re using different library versions. Like if my scenario led to something like A_2.13 v 1.1 vs. A_3 v 1.2 then you’d potentially get errors from a mismatch b/t the 1.1 and 1.2 versions of the library. But that’s just the same old version incompatibility / classpath issue we deal with all the time.

But, assuming you don’t have any macros involved, and assuming that the library in question isn’t doing anything sketchy with their versioning (like publishing different source code into the same lib version on different Scala version suffixes), I would have thought that A_2.13:1.1 and A_3:1.1 would be interchangeable at runtime.

1 Like
4

Take this example from the cats library. It has some code that differs between 2.x and 3.x. I’m pretty sure the binary signatures of those methods are different in the Scala 3 version, because the parameter lists are in a different order.

2 Likes
5

Wow, that is an interesting example. I wonder why they moved all of the using parameter lists to the front?

I take your point, though, I guess there’s nothing stopping some library using the src/main/scala2.x/3.x convention to provide completely incompatible APIs or implementations depending on the versions. We might hope that isn’t the case in practice (and maybe sometimes you get lucky and things “just work”) but it’s always a risk.

So I guess the takeaway here is that even though you can safely mix and match 2.13 and 3.x deps in your classpath, you effectively have to treat them as 2 independent dependency trees. And if you end up with any overlap between your 2.13 and 3.x tree, you’re going to be stuck basically.

1 Like
6

I did some digging and apparently it’s because of this issue. That should be mostly source compatible though (but not binary compatible), because of how Scala 3 treats using parameters differently than implicit parameters.

Even without the obvious cases like this one I’m not sure if it’s guaranteed that Scala 2 and 3 always compile the same source to exactly the same binary method signatures. But we’d probably need some input from a Scala 3 maintainer to know for sure.

Edit: smarter says

No, they might not be binary-compatible
in particular because of lampepfl/dotty#11808 and lampepfl/dotty#11846

4 Likes
7

Awesome, thanks again for the digging and sharing those links. It makes sense why all this is the case, I think I had just been overly optimistic about how the 2.13/3.0 interop story would play out. Hopefully this post will help anyone else who is similarly confused. I will probably just wait for a few more of my deps to upgrade so that I can keep the 2.13 vs. 3.0 dep trees completely unentangled.

1 Like
8

  1. If this is even mentioned on the Scala 3 migration page from the point of view of a library user, I did not see it. Thought I had waited long enough to try migration the other day, but without warning hit this same issue. This thread was helpful, but please update the migration page with info about this risk, and if possible some useful guidance.

  2. I use sbt in a fairly elementary way. Maybe I did something wrong, but adding “ThisBuild / evictionErrorLevel := Level.Info” just above the libraryDependencies in build.sbt seemed to have no effect. I’m not sure I really want to just ignore these errors anyway, but would like to understand why that did not work.

  3. Earlier was getting output that at least showed sources of transitive dependency conflicts, but now don’t know how to even get that info back again, either.

FYI, errors look like this:

[error] org.scodec:scodec-bits _3, _2.13
[error] co.fs2:fs2-core _3, _2.13
[error] org.scala-lang.modules:scala-collection-compat _3, _2.13
[error] org.log4s:log4s _2.13, _3
[error] com.lihaoyi:sourcecode _2.13, _3
[error] org.typelevel:cats-effect _3, _2.13
[error] io.circe:circe-core _2.13, _3
[error] io.circe:circe-jawn _2.13, _3
[error] org.typelevel:simulacrum-scalafix-annotations _3, _2.13
[error] org.typelevel:cats-kernel _3, _2.13
[error] co.fs2:fs2-io _3, _2.13
[error] org.http4s:http4s-core _2.13, _3
[error] org.typelevel:jawn-parser _2.13, _3
[error] org.typelevel:cats-core _3, _2.13
[error] io.circe:circe-numbers _2.13, _3

1 Like
9

Hello, you may find help with 2.13/3 compatability using the new cross version features in sbt 1.5, you can read about them in this scala-lang.org blog: Scala 3 in sbt 1.5 | The Scala Programming Language

10

Thanks, was already using that when errors occurred. This thread was actually devoted to cases where that new mechanism does not prevent issues with inconsistent transitive dependencies from different libraries. For now I’ve re-stabilized by backing off to Scala 2.13, keeping same library versions I was trying to use with Scala 3.

11

I confirm that in my case also, transitive dependencies have lead to the mentioned suffix conflict (Modules were resolved with conflicting cross-version suffixes in ProjectRef...), and removing the ... cross CrossVersion.for3Use2_13 appendix from one libraryDependencies += ... line could solve an issue today. CrossVersion.for3Use2_13 is not “cheap”, it has a high price as soon as a single arbitrary, subordinated component is used in another place where the decision for _2_13 vs. _3 is chosen differently, since there is only “room” for one _2_13 or _3 exclusively as by component. sbt dependencyTree >deps.txt helped my with the investigations, and finally i’ve drawn I diagram, a Goez-Into-Graph, by hand, with many arrows :wink: Heureka! Thank You all!