Hello
In my spark job, b/c I don’t like to write the main() method, I use the convenient way of :
object Myobj extends App { }
But this sometimes works, sometimes not. When it doesn’t work, I remove the extends and add a main(), it then works.
Do you know why? Thank you
Just forget that App exists, it was a noble effort but a bad idea.
When you say it “doesn’t work”, do you get an error message, other unexpected behavior, or what? If it’s an error message, include the complete text of the error.
Even the official documentation of Spark says not to use App because it may not work: Quick Start - Spark 3.5.0 Documentation
Note that applications should define a
main()method instead of extendingscala.App. Subclasses ofscala.Appmay not work correctly.
Now, why that is the case? I would guess that the weird compile-time black magic of DelayedInit doesn’t play well with the weird runtime back magic of Spark reflection.