NullPointer Exception

Hi All,

Please could someone help on this below issue.

Getting error as below in spark 2.3.0

java.lang.NullPointerException +details java.lang.NullPointerException at org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter.write( at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.serializefromobject_doConsume$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.mapelements_doConsume$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.deserializetoobject_doConsume$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)

Getting above error while we are trying to move from spark 2.1.1 to 2.3.0 . The same code works well in 2.1 but we are getting this error surprisingly with 2.3 .

Any help is greatly appreciated

Hi, can you post the exact code snippet for this bug?

Looks like a problem with generated code passing null to this function.

possibly related

Probably you have a column that is nullable, but Spark thinks it’s not nullable, so it doesn’t generate a null check.