Dear All,
I have a large collection of objects which I am trying to make into a set. When doing this using the .toSet method (in scala 2.13.1) I get a stack overflow error. What is the recommended way around this?
One way I can think of is to group, make each group a set and tail recursively form a union. Can someone recommend a better way (or does this look like a reasonable way)?
regards,
Siddhartha
Uhm that is weird.
AFAIK, most operations of the stdlib are implemented using variables and while loops.
I just tested and can’t reproduce the problem like this:
LazyList.from(1).take(1000000).toSet // works!
LazyList.from(1).take(10000000).toSet // OutOfMemoryError: Java heap space.
So, I guess I would have blowup the stack before the heap.
Can you create a reproducible example?
Which collection are you using?
It was fixed in my case. Perhaps a factor is that the objects I used were themselves deeply nested tree-like structures, and it was in computing their hashes that it crashed.
I replaced toSet by grouping, using toSet on groups and tail recursively taking the union, and also changed my internal representation so that Sum(a, Sum(b, Sum(c, ..))). became Sum(Vector(a, b, c, ...) (here a, b etc are themselves complicated objects). I do not know which change fixed it, and the crash happens after over an hour of computations, so cannot be replicated easily.
The structure I was using is List I believe (I used groupMapReduce to get a Map and then a function of the form m.map{case (k, v) => fn(k, v).
regards,
Siddhartha