Hi,
I have two columns,
A and B.
a1 b1
a1 b1
a1 b2
a2 b1
a2 b3
a3 b3
a3 b4
I need to aggregate and group A and B and create new columns in scala dataframe, so results should be,
> A B => a1 a2 a3
------------------------------------------------------------------------ a1 b1 b1 b1 b3 a1 b1 b1 b3 b4 a1 b2 b2 null b3 a2 b1 a2 b3 a3 b3 a3 b4 a3 b3
a1,a2 and a3 columns (derived from A and B) as below,
a1 a2 a3
------------------------------------------------------------------------
b1 b1 b3
b1 b3 b4
b2 null b3
a1I tried with pivot and aggregation, but it remoes duplicate entries from column a1,a2,a3.
How can I achieve the above using scala spark.
Thank you.