Nested parallel flatmaps

Question
1

I have some code of the form:

val expensive = (1 to 10).par.map(longOperation)

and the use of .par yields a considerable performance benefit. Fantastic.

(As an aside, @SethTisue, that’s an example of a win using Scala).

I’m notnow fiddling around with something like:

val moreExpensive = (1 to 10).par.flatMap{input =>
                           // ^^^
    val fragments: Seq[Int] = fragmentsFrom(input)
    fragments.par.map(anotherLongOperation)
           // ^^^
}

Am I correct in expecting there to be more potential opportunities for thread parallelisation using the nested par-in-flatmap?

A quick look at the implementation seems to indicate that the parallel version of flat-mapping doesn’t know whether the underlying collection is parallel or sequential (although that could be hidden further down).

So I’m wondering of a) there is an opportunity for further parallelisation here in the nested mapping and b) whether it is entirely independent of what is going on in the outer flat-mapping?

1 Like
2

Launching more threads not necessarily will make this faster, it may actually make it slower.
I personally would keep the parallelism in a single level, whichever makes more sense; usually the outer one.

However, as usual with performance, the only right answer is that you need to profile and benchmark yourself.

3

Ah yes, I’ve heeded the thundering-herd health warning thus far. :smile:

Nevertheless, I’m interested as to whether there is any potential increase in parallelism, and whether that might be coordinated between the two levels via the flat-map, or simply treated as a Cartesian product of threads stolen from the fork-join pool.

For the full story: Further performance improvements. · Issue #35 · sageserpent-open/kineticMerge · GitHub

However, these days I can’t chest press so much as I used to - too much hacking at the keyboard, must try harder. Lat pull downs are better, I reckon I could beat 50 kg on a good day. Christmas is going to take its toll on my side-profile, too. :laughing:

Cheers!

1 Like
4

I remember there was an issue about how to execute the nested tasks.

Sorry I don’t have the ticket at hand; not sure if it is now scala/scala or on the parallel-collections.

Edit: TaskSupport respects specific FJP by som-snytt · Pull Request #6159 · scala/scala · GitHub

and two tickets with discussion

https://github.com/scala/bug/issues/10577

and

https://github.com/scala/bug/issues/11036

The second ticket is the one I remember for its length.

I’d forgotten this comment:

The architecture makes me feel like I need a “conspiracy wall”.

1 Like
5

The workout you’re looking for is “French press” in the morning and “12 ounce curls” in the afternoon. In the U.K. it might be a pint.

1 Like
6

Thanks folks for the suggestions (and TIA if others want to chip in) - I now have some light reading this weekend.

I’m going to try flattening upfront and then running it all through one level of .par.

Also, es geht um Gewicht - dann schlage ich vor: um kraeftiger zu sein, trink eine Flasche Wein, aber fuer groesstene Masse, trink direkt vom Fass.

My German is pretty rusty these days, as you can tell.