Can we count how many times FileUtil.copy is triggered as part of foreachPartition and foreach?

Take this code:

if (filesToCopy.size > 0){
spark.sparkContext.parallelize(filesToCopy).foreachPartition { rows =>
rows.foreach { file =>

      val fromPath = something
      val targetFileName = something
      val toPath = something
      val fromFs = something
      val toFs = something

      var magic = a_condition

      if (magic == false){
        FileUtil.copy(fromFs, fromPath, toFs, toPath, true, conf)
      }
    }
  }
}

Is it possible to track how often FileUtil.copy was used, or how often the variable magic was set to true?

You can use a variable called magicCount at top level or in an object or an appropriate companion object:

      val magic = a_condition

      if magic then magicCount += 1
      else FileUtil.copy(fromFs, fromPath, toFs, toPath, true, conf)

I have done just that. When I try to return the magicCount it is always 0.
The files are being moved properly, so the rest of the code is working as intended.

Have you verified that “magic” is actually true in one or more cases? You can use a simple print statement or an assert statement for that.

Yes I have. I was able to see the correct files being moved.
The following solution worked:

val magicCount = spark.sparkContext.longAccumulator(“SumAccumulator2”)
and if magic is true:
magicCount.add(1)

I’m glad to hear that you found a workable solution. What is the type of magicCount?