WriteStream is not a member of org.apache.spark.sql.DataFrame

Question
1

Hi
I am trying to stream a log file from HDFS in the spark shell. The readStream is generating a data frame but in the process of writing the stream i am getting the following error

:27: error: value WriteStream is not a member of org.apache.spark.sql.DataFrame

scala> var lines = spark.
| readStream.
| text("/tmp/messages")
lines: org.apache.spark.sql.DataFrame = [value: string]

scala> lines.
| WriteStream.
| queryName(“lines”).
| outputMode(“append”).
| format(“memory”).
| start
:27: error: value WriteStream is not a member of org.apache.spark.sql.DataFrame

I am using
Scala version 2.11.12
Spark version 2.3.2.3.1.4.0-315

2

The method is called writeStream with a lowercase “w”. In Scala methods usually start with a lowercase letter and classes with an uppercase letter.

1 Like