NullPointer Exception

Question
1

Hi All,

Please could someone help on this below issue.

Getting error as below in spark 2.3.0

java.lang.NullPointerException +details java.lang.NullPointerException at org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter.write(UnsafeRowWriter.java:194) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.serializefromobject_doConsume$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.mapelements_doConsume$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.deserializetoobject_doConsume$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)

Getting above error while we are trying to move from spark 2.1.1 to 2.3.0 . The same code works well in 2.1 but we are getting this error surprisingly with 2.3 .

Any help is greatly appreciated

2

Hi, can you post the exact code snippet for this bug?

3

Looks like a problem with generated code passing null to this function. https://github.com/apache/spark/blob/v2.3.0/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/codegen/UnsafeRowWriter.java#L194

possibly related https://github.com/datastax/spark-cassandra-connector/issues/1062

Probably you have a column that is nullable, but Spark thinks it’s not nullable, so it doesn’t generate a null check.