I know there are a lot of instance methods in spark which have the lazy computing capability. But for the theory in scala, can you help giving the examples to show what’s the lazy computing and why need them?
Thank you.
I know there are a lot of instance methods in spark which have the lazy computing capability. But for the theory in scala, can you help giving the examples to show what’s the lazy computing and why need them?
Thank you.
What do you mean with “lazy computing”? You mean lazy vals
? By name parameters, i.e. => Foo
? Or effect types like IO
which are sometimes referred as lazy futures?
such as Iterator is a lazy object which can be evaluated by needed.
In Spark, it is used to reduce network operations to avoid latency as much as possible (the biggest bottleneck when your huge dataset is distributed across multiple computers over a network). I highly recommend taking the Big Data Spark course https://www.coursera.org/learn/scala-spark-big-data/
Here is a slide from the course:
map
, filter
, flatMap
, groupBy
etc. They return collections from collections.reduce
, fold
, aggregate
etc. They return single values from collections.
I am skilled at spark. what I want to learn is the native lazy implementation in scala language. thank you.
In that case you should take this course https://www.coursera.org/learn/scala-functional-program-design which covers Lazy Evaluation (native Scala) in detail (week 2), it even has a programming assignment that uses infinite lazy streams.
Uhm, I am still not sure what is the answer you want.
But, use cases for the laziness of Iterator
we have: