I have two implementations of a function, and I’d like to get some statistics about which one is faster. My instinct is to make sure the garbage-collector runs before I start each of them. I.e., I don’t want the cost of garbage collecting implementation-A to be tallied into the computation of implementation-B.
Is that a valid concern? Is there something I should do to make sure I don’t count the GC of implementation-A in the timing code of implementation-B?
For benchmarking single functions, I’d recommend the Java Microbenchmarking Harness (JMH). It doesn’t require a lot of setup and manages all the stuff that one usually does forget when benchmarking on the JVM, like GC, JIT-Warmup and including the timing functions in the measurement. It also prints nice statistics at the end.
There’s also an sbt plugin, see this blog post for an example of using JMH with scala.
Of course, garbage collection is major concern for benchmarking. If garbage collection is triggered inside a method, that method will take a much longer time to complete. On the other hand, excluding garbage collection does not seem like a fair assessment - what if one method runs faster, but produces more garbage that takes more time to collect later?
I’m actually suspicious of microbenchmarking in general, because you never know what kind of optimizations the JVM may attempt. I would benchmark by running the method many times, with at least a few minutes total run time. Try that a few times to assess variations. Also, make sure not to ignore the return value to prevent the method call from being optimized away - I usually take the hash code of the return value and add all the hash codes.
Best, Oliver
Yes, that’s exactly the problem. If I have to functions f1, and f2, and f1 produces a lot of garbage, and the gc is triggered in f2. The the benchmarking will indicate that f2 is slower. To resolve this I’d like to trigger the gc before starting the benchmark timer on f2.
Maybe I wasn’t clear. Let’s say you prevent garbage collection from interfering with your benchmark and you find that f2 is 10 milliseconds faster per call than f1. But f2 produces more garbage than f1, and let’s say the garbage f2 produces per call takes 20 milliseconds longer to clean up than the garbage of f1. Then, what good was it that f2 finished 10 milliseconds faster?
The JVM garbage collector in general is just not that controllable. The only standard “control” mechanism in the whole Java API (that I know of) is System.gc(), which basically is only for sending a hint to the JVM that now would be a good time to run GC. The JVM does not guarantee anything about the behavior of System.gc(), however. It might run some GC, it might not. Objects that are unreachable may be finalized and their memory reclaimed, or they might not. It is up to the JVM implementation to decide what it wants to do.
Brian Maso