Hi to all community,
This is my first post, and I need a little help, in a scala programming task, that is not so trivial (at least for me).
I’m using scala in ver 2.10, under a Spark 3.0.0-preview2 versions.
Imported from a mysql DB, my datas are of this type:
95,118.37,118.47,111.725,114.3,1049181,AMP,2020-04-14
96,116.88,117.84,113.11,114.92,827085,AMP,2020-04-13
97,113.64,124.61,113.64,120.47,1608575,AMP,2020-04-09
98,104.48,112.48,102.28,111.69,996230,AMP,2020-04-08
99,109.17,112.23,102.41,103.48,1302910,AMP,2020-04-07
100,42.25,42.25,41.73,41.82,639964,G,2020-08-26
101,41.98,42.15,41.76,42.12,501219,G,2020-08-25
102,41.52,42.015,41.45,41.9,479076,G,2020-08-24
103,41.27,41.46,40.99,41.16,752730,G,2020-08-21
104,41.74,41.965,41.25,41.3,596435,G,2020-08-20
105,42.14,42.21,41.87,41.94,422493,G,2020-08-19
Then, with a mapping process, those datas get reformatted in a Tuple2<Key,Value> of this type:
(AMP,(1,156.77,156.915,155.03,155.74,527938,AMP,2020-08-26))
(AMP,(2,159.48,159.88,156.86,156.99,535905,AMP,2020-08-25))
(AMP,(3,155.38,157.75,155.33,157.72,758272,AMP,2020-08-24))
(AMP,(4,155.24,156.79,153.92,154.51,653496,AMP,2020-08-21))
(AMP,(5,155.24,157.39,154.27,155.14,516138,AMP,2020-08-20))
(AMP,(6,156.65,160.06,156.57,156.85,577637,AMP,2020-08-19))
(AMP,(7,158.05,158.35,156.34,156.5,544429,AMP,2020-08-18))
(AMP,(8,159.69,159.82,157.76,157.83,437624,AMP,2020-08-17))
Where, every single record is of the type:
org.apache.spark.rdd.RDD[(String, (Int, Double, Double, Double, Double, Int, String, String))]
Then, I need to group all the keys, and wrote a “groupByKey” procedure:
val SA = Simboli.groupByKey
Resulting in a variable:
org.apache.spark.rdd.RDD[(String, Iterable[(Int, Double, Double, Double, Double, Int, String, String)])]
My question is now: Can I create a new variable of type “Vector”, or “Sequence”, inserting every single record of this strange type, in the vector list?
For example, a vector where every single item is a new:
RDD[(String, Iterable[(Int, .....
The only way I found is to transform this kind of variable in this way:
- take only the first “group”
val SAG : Array[(String, Iterable[(Int, Double,
Double, Double,
Double, Int,
String, String)])] = SA.take(1);
Extract the “Iterable” part:
val SAGITB : Array[Iterable[(Int, Double,
Double, Double,
Double, Int,
String, String)]] = SAG.map(item => item._2);
Convert “Iterable” in “Iterator”:
val SAGITT : Array[Iterator[(Int, Double,
Double, Double,
Double, Int,
String, String)]] = SAGITB.map(item => item.iterator);
Extract values:
val SARDD : Array[(Int, Double,
Double, Double,
Double, Int,
String, String)] = SAGITT.map(item => item.next);
Finally, I’m trying to populate a Vector, or Sequence, with every single item, insiede a for loop, but I can’t. This is my last attempt:
val SV3 : Vector[Array[(Int, Double, Double,
Double, Double, Int,
String, String)]] = Vector.empty;
for (it <- 0 to 20){
println("Riga numero: " + it);
SV3 :+ SAGITT.map(item => item.next);
}
Finally my question is: How can I populate a Vector, or Sequence, with datas of type “iterable”, or “iterator” or, otherways, How can I extract all datas from an RDD of Iterables, convert and populate with these datas a simple Vector?
Thank you so much !!