A use case of implicit class

Documentation
1

in spark-shell (it’s actually scala REPL):

scala> val list = List(3,2,1,4)
list: List[Int] = List(3, 2, 1, 4)

scala> val df = list.toDF("number")
df: org.apache.spark.sql.DataFrame = [number: int]

scala> df.show
+------+
|number|
+------+
|     3|
|     2|
|     1|
|     4|
+------+

As you see the List can be converted into Dataframe directly, though scala has no this native method: toDF().

This is b/c spark developers use the implicit class to add this method secretly to Seq class.

While pyspark doesn’t have this implementation:

>>> list = [3,2,1,4]
>>> list.toDF("number")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'list' object has no attribute 'toDF'