(Long) compile times with Type derivation (Scala 3)

Question
1

I struggle with long compile times.
At the moment I think the problem is the Type derivation for JSON marshalling.

So I created a simple project that lets me generate some case classes and enums.

See: GitHub - pme123/scala3-compile-tests: I struggle with long compile times. This project should help optimizing it and ask questions to specialists;).

It looks like circe takes twice as long as jsoniter.

I tested with 200 generated classes.

Results Circe:

[info] Sourcefile               Lines   Tokens   Tasty  Complexity/Line    Directory
[info] JsoniterGenerator.scala     28      314      59   2.11  moderate    /Users/mpa/dev/experiments/scala3-compile-tests/src/main/scala/scala3/compile/tests
[info] CirceGenerator.scala        26      316      60   2.31  moderate    /Users/mpa/dev/experiments/scala3-compile-tests/src/main/scala/scala3/compile/tests
[info] Tester.scala                59      520     260   4.41  moderate    /Users/mpa/dev/experiments/scala3-compile-tests/src/main/scala/scala3/compile/tests/generated
[info] Generated6.scala            66      442     512   7.76  high        /Users/mpa/dev/experiments/scala3-compile-tests/src/main/scala/scala3/compile/tests/generated
.....
[info] -----------------------------------------------------------------
[info] Total                     3347    22808   25451   7.60  high
[info] done compiling
[success] Total time: 68 s (01:08), completed 17 Sep 2023, 13:34:31

Results Jsoniter:

[info] Sourcefile               Lines   Tokens   Tasty  Complexity/Line    Directory
[info] JsoniterGenerator.scala     28      314      59   2.11  moderate    /Users/mpa/dev/experiments/scala3-compile-tests/src/main/scala/scala3/compile/tests
....
[info] Generated11.scala           63      422     476   7.56  high        /Users/mpa/dev/experiments/scala3-compile-tests/src/main/scala/scala3/compile/tests/generated
[info] Tester.scala               204     2023    1076   5.27  high        /Users/mpa/dev/experiments/scala3-compile-tests/src/main/scala/scala3/compile/tests/generated
[info] -----------------------------------------------------------------
[info] Total                    12795    86631   95890   7.49  high        
[info] done compiling
[success] Total time: 32 s, completed 17 Sep 2023, 13:40:07

So here my Questions:

  • is there a way to decrease the compile time?
  • are there Json Libraries that have faster compile times?

(Mac 2.6 GHz 6-Core Intel Core i7 / 32 GB 2667 MHz DDR4)

2

Are you auto-deriving? If so go to semi-deriving. Took a project from 5 minute compile times down to under a minute by avoiding auto-deriving.

3

Yes, I use semi-deriving, the link shows you such an implementation for circe: https://github.com/pme123/scala3-compile-tests/blob/master/src/main/scala/scala3/compile/tests/generated/Generated10.scala

5

is there a way to decrease the compile time?

As far as I am aware, the only things you can do are:

  1. ensure that each instance is only derived once - if your code is structured as in your test project, then you’re already doing that.
  2. only derive what you really need - for example, circe has separate Encoder and Decoder type classes and since in circe, derivation of a Codec involves derivation of the corresponding Encoder and Decoder instances, you might be able to reduce compiltation time (maybe by about 50%) if you only derive Encoder or Decoder instead of Codec (of course: only in case you actually only need one of the two).

Additionally, there are various things one can do to reduce unneccessary re-compilation (e.g. put the your “protocol” classes (types and their codecs) into separate files so that they do not get re-compiled if unrelated code is changed), but I guess that is not what you are looking for.

are there Json Libraries that have faster compile times?

I am not aware of any comparison of Scala 3 Json libraries regarding the compilation time (for derived codecs) nor have a seen any of the libraries pointing out that aspect as one of its advantages.

More generally, I have not yet seen any publication (of talks etc) regarding how to optimize Scala 3 type class derivation regarding compilation times (though I noticed that those times can significantly vary depending on how the derivation mechanism is implemented).

2 Likes