Adding new column with custom text to Dataframe

Question
1

How can i create a multi column check and create a comma separated string and get it added as a new column at the end of a dataframe. Ex I need to check if i have a List(22,33,44,55,66,77). The NewCol should be updated as below.

image

Example:
val list = List(4,5,10,7,2)
val df = List(“a”,“b”,“c”,“d”,“e”).toDF(“row1”)

df.collect()
.map(_.getAsString)
.zip(list).toList
.toDF(“row1”, “row2”)

The above example very much works for me. But if my df is multi columns how can get the new df with all columns in first df appended with row2.

2

Why are you doing a collect at the beginning? You will lose your parallel processing if you do that.

3

You can use the spark version of all of these method calls to keep everything with spark. For example, you have spark zip here http://spark.apache.org/docs/latest/api/java/org/apache/spark/rdd/RDD.html#zip-org.apache.spark.rdd.RDD-scala.reflect.ClassTag-