Can we count how many times FileUtil.copy is triggered as part of foreachPartition and foreach?

Question
1

Take this code:

if (filesToCopy.size > 0){
spark.sparkContext.parallelize(filesToCopy).foreachPartition { rows =>
rows.foreach { file =>

      val fromPath = something
      val targetFileName = something
      val toPath = something
      val fromFs = something
      val toFs = something

      var magic = a_condition

      if (magic == false){
        FileUtil.copy(fromFs, fromPath, toFs, toPath, true, conf)
      }
    }
  }
}

Is it possible to track how often FileUtil.copy was used, or how often the variable magic was set to true?

2

You can use a variable called magicCount at top level or in an object or an appropriate companion object:

      val magic = a_condition

      if magic then magicCount += 1
      else FileUtil.copy(fromFs, fromPath, toFs, toPath, true, conf)
3

I have done just that. When I try to return the magicCount it is always 0.
The files are being moved properly, so the rest of the code is working as intended.

4

Have you verified that “magic” is actually true in one or more cases? You can use a simple print statement or an assert statement for that.

5

Yes I have. I was able to see the correct files being moved.
The following solution worked:

val magicCount = spark.sparkContext.longAccumulator(“SumAccumulator2”)
and if magic is true:
magicCount.add(1)

6

I’m glad to hear that you found a workable solution. What is the type of magicCount?