Hi,
I have an dataframe table as follows,
+--------------------+--------------------+--------------------+
| VALUE | COL1 | COL2 |
+--------------------+--------------------+--------------------+
|TIME_STAMP,2016-1...| TIME_STAMP| 2016-12-04 11:56:10|
|TIME_STAMP,2016-1...| TIME_STAMP| 2016-12-08 12:41:13|
|TIME_STAMP,2016-1...| TIME_STAMP| 2016-10-20 12:34:55|
|TIME_STAMP,2017-1...| TIME_STAMP| 2017-06-12 13:29:59|
| ANSWER,YES| ANSWER| YES|
| ANSWER,YES| ANSWER| YES|
| ANSWER,NO | ANSWER| NO |
| ANSWER,YES| ANSWER| YES|
| ANSWER,NO | Result| NO |
| BOOK_TYPE,SC.| BOOK_TYPE | SCIENCE|
| BOOK_TYPE,MA.| BOOK_TYPE | MATHS|
| BOOK_TYPE,SC.| BOOK_TYPE | SCIENCE|
| BOOK_TYPE,EV.| BOOK_TYPE | EVS|
| SELF_ID,2..| SELF_ID| 200|
| SELF_ID,4..| SELF_ID| 400|
| SELF_ID,3..| SELF_ID| 300|
| SELF_ID,2..| SELF_ID| 200|
| SEARCH_OUTCOME,PASS| SEARCH_OUTCOME| PASS|
| SEARCH_OUTCOME,PASS| SEARCH_OUTCOME| PASS|
| SEARCH_OUTCOME,FAIL| SEARCH_OUTCOME| FAIL|
| Reason,NA| Reason| NA|
| Reason,NA| Reason| NA|
| Reason,NA| Reason| NA|
| Reason,AV| Reason| AV|
+--------------------+--------------------+--------------------+
I want to convert this dataframe tabe to table 2 as follows,
+--------------------+--------------------+--------------------+
| TIME_STAMP | ANSWER| BOOK_TYPE |etc....
+--------------------+--------------------+--------------------+
|2016-12-04 11:56:10 | YES | SCIENCE |
|2016-12-08 12:41:13 | YES | MATHS |
|2016-10-20 12:34:55 | NO | SCIENCE |
|2017-06-12 13:29:59 | YES | EVS |
| | NO | YES|
+--------------------+--------------------+--------------------+
How can I convert from table 1 to table 2 using spark dataframe and scala.
Thank you