How to use Monix to execute very large set of CPU intensive tasks



I am looking for advice on what is the best (better?) approach to use the Monix library. I have looked at the documentation but the API is extensive and I don’t have a handle on its structure and how best to use it. Would like some advice on how to solve the following problem:

I have a set of tasks that are executed according to a number of parameters that are passed on to that task. The number of tasks to execute is large and the parameters are made available via a lazy stream. I would like to go through this stream and take the parameter to execute the task in parallel. On completion the task returns a list of numbers which I want to store in a (text) file.

The aim here is maximize CPU usage. Would using a Consumer using a load balancer be the better solution? In this case can I still select a scheduler (example ForkJoin) to maximize CPU use?

Appreciate any additional pointer on this.



I am no expert on any of this, but have been using Monix recently, and find that Task.gather and do use most of my CPUs.



Thanks Siddhartha. I did considered this but the Seq of tasks I have is lengthy and this may be an issue (memory). Can you give me an idea of the number of tasks you execute and if you used any specific scheduler?