data sample is in csv and i have to read and write as parquet
id_a,id_b,temp,TRF
1000391,10003421,{“decider”:false,“Vectrums”:[0,0,0,1,1,1,1,0]},FALSE
100090441,1000091555,{“decider”:false,“Vectrums”:[0,0,0,1,1,1,0,0]},FALSE
schema:
id_a:string
id_b:string
temp:struct
decider:boolean
Vectrums:array
element:integer
trf:boolean
i am trying with following schema:
val structSchema=StructType(StructField(“temp”,StructType)
(StructField(“decider”,BooleanType(),True),
(StructField(“Vectrums”,ArrayType(IntegerType()),True)
)),StructField(“id_a”,StringType(),True),
StructField(“id_b”,StringType(),True),
StructField(“TRF”,StringType(),True))
error:
not found: value BooleanType
not found: value ArrayType
tried to import scala datatypes
expectation: Need to read the data which is in csv and write to parquet.how can i apply schema and read , please suggest if there is any sample code or outline for this kind of scenario