Query execution eats all my memory for Big Data

Hi EveryOne,

I have a db2 table that is having 1.5 million records and while querying only have couple of where conditions, but the results are taking a lot of time to fetch. How should i optimize?
What all things i can do to improve performance.

This seems like more of a db2 question than a Scala question – I suspect you’re going to have more luck asking elsewhere.

That said – how many rows are you fetching? I don’t know db2 at all, but that’s the first thing I would look at if you’re running out of memory. You may be trying to fetch more records than can be reasonably handled that way, in which case you probably need to look into a streaming-oriented solution…

1 Like

I too have been asked to take up streaming, so that a small part of data is processed at a time. But would it be the only option left? I mean no other way like RDD, parallelism, partition, tempview, thread, cache, any specific way the sql query is executed/called. other spark configuration, memory, driver-memory, executor-cores, executor-memory, executor-cores etc. I haven’t tried all of them but an clue will be of much help.
My environment is scala running on the top of spark.