If I have a sequence of objects, and a sequence of transformations of object to object, Is there an idiom I can use to run the transformations in parallel?
More specifically suppose I have a seq:Seq[A1], and functions f1:A1=>A2, f2:A2=>A3, … fk:Ak=>Ak+1,
E.g., How can I do the following:
val result:Seq[T6] = seq.map(f1).map(f2).map(f3).map(f4).map(f5)
in a way so that the functions f1…fk have a chance to run in parallel/concurrently if resources permit?
I suspect you’re looking for the parallel collections library, which was split out from stdlib in 2.13. Specifically, I suspect you want ParSeq.
(NB: I haven’t used this stuff myself. But what you’re asking seems to match those docs.)
@jducoeur, as I understand a ParSeq allows one operation such as 'foldLeft` to partition the sequence and perform on it in parallel. What I want is different, at least I think it is different. I want a sequence of non-parallelizable operations to be done concurrently.
val result:Seq[T6] = seq.map(f1).map(f2).map(f3).map(f4).map(f5)
should be semantically the same as seq.map(f5.compose(f4.compose(f3.compose(f2.compose(f1)))))
Is there some sort of conversion guide that explains how to convert from parallel collections < 2.13 to newer? Is it just changing the include path? As I recall it is more than that. Right?
ParSeq means that if you do map(f1), f1 can be applied to multiple objects in parallel to build a new (parallel) collection.
If you do map(f1).map(f2), it will first apply f1 to all objects and build a new collection before it starts applying f2.
If you want to begin applying f2 before all it is finished applying f1 to all objects, you could compose f1 and f2, or use views, or use some library. I think ScalaZ and cats support that. Spark, of course, supports it, too, but would be an overkill unless you want to do distributed computation.
Yes, composing would be one possibility. Although, unless I’m confusing myself, composing doesn’t work if we replace map with flatMap as each stage would be a one-to-many mapping.
I’ll take a look at views. Not sure what that is yet.