I have an ASCII (text) data file that is around 2 GB in size. I am trying to read it line by line and save a reduced version of it. Something like this:
for (line <- io.Source.fromFile(fileName).getLines) println(line)
Actually, I am reducing the number of lines and the size of each line, but that is irrelevant here. When I run it, I get:
Exception in thread “main” java.lang.OutOfMemoryError: Requested array size exceeds VM limit
I assume this is happening because the entire file is getting loaded into memory. Is there a simple way to read and process one line at a time without loading the entire file? Thanks.