How to use Monix to execute very large set of CPU intensive tasks

Question
1

Hello,

I am looking for advice on what is the best (better?) approach to use the Monix library. I have looked at the documentation but the API is extensive and I don’t have a handle on its structure and how best to use it. Would like some advice on how to solve the following problem:

I have a set of tasks that are executed according to a number of parameters that are passed on to that task. The number of tasks to execute is large and the parameters are made available via a lazy stream. I would like to go through this stream and take the parameter to execute the task in parallel. On completion the task returns a list of numbers which I want to store in a (text) file.

The aim here is maximize CPU usage. Would using a Consumer using a load balancer be the better solution? In this case can I still select a scheduler (example ForkJoin) to maximize CPU use?

Appreciate any additional pointer on this.

TIA

2

I am no expert on any of this, but have been using Monix recently, and find that Task.gather and Task.zip do use most of my CPUs.

regards,
Siddhartha

3

Thanks Siddhartha. I did considered this but the Seq of tasks I have is lengthy and this may be an issue (memory). Can you give me an idea of the number of tasks you execute and if you used any specific scheduler?

TIA