Take this code:
if (filesToCopy.size > 0){
spark.sparkContext.parallelize(filesToCopy).foreachPartition { rows =>
rows.foreach { file =>
val fromPath = something
val targetFileName = something
val toPath = something
val fromFs = something
val toFs = something
var magic = a_condition
if (magic == false){
FileUtil.copy(fromFs, fromPath, toFs, toPath, true, conf)
}
}
}
}
Is it possible to track how often FileUtil.copy was used, or how often the variable magic was set to true?
Russ
June 29, 2023, 5:10pm
#2
You can use a variable called magicCount at top level or in an object or an appropriate companion object:
val magic = a_condition
if magic then magicCount += 1
else FileUtil.copy(fromFs, fromPath, toFs, toPath, true, conf)
I have done just that. When I try to return the magicCount it is always 0.
The files are being moved properly, so the rest of the code is working as intended.
Russ
June 30, 2023, 5:07pm
#4
Have you verified that “magic” is actually true in one or more cases? You can use a simple print statement or an assert statement for that.
Yes I have. I was able to see the correct files being moved.
The following solution worked:
val magicCount = spark.sparkContext.longAccumulator(“SumAccumulator2”)
and if magic is true:
magicCount.add(1)
Russ
July 3, 2023, 7:09pm
#6
I’m glad to hear that you found a workable solution. What is the type of magicCount?