How to subscribe to kafka topic

Question
1

Hi,

I’m using spark 2.4 . I’m new to scala programming . I want to use structure streaming and subscribe to a kafka topic . Please let me know the steps associated with it. I tried with the following but getting an error. Please help in resolving the error.

val df = spark.readStream.format(“kafka”).option(“kafka.bootstrap.servers”, “***:”).option(“subscribe”, “RP_LOGS”).load()
org.apache.spark.sql.AnalysisException: Failed to find data source: kafka. Please deploy the application as per the deployment section of “Structured Streaming + Kafka Integration Guide”.;
at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:652)
at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:161)
… 49 elided

2

This is a list for the Scala language, and not for Apache Spark. In the future, you will get faster responses to questions like this from the Apache Spark community.

With that said, your problem is likely that you do not have the spark-sql-kafka dependency on your classpath at runtime. You either need to specify it as a compile-time dependency and build it into your jar using the sbt-assembly or sbt-native-packager plugins, or you need to use the --packages option for spark-submit or spark-shell.