05 — Benchmarking with JMH

Here is an uncomfortable truth: almost every hand-rolled Java microbenchmark you have ever seen is wrong. The five-line main with System.nanoTime() around a loop is not measuring what its author thinks. The JIT hasn’t warmed up on the first iteration, dead-code elimination erased the loop body after iteration two, constant-folding collapsed your computation into a literal at compile time, and the numbers you’re reading are somewhere between meaningless and actively misleading.

JMH — the Java Microbenchmark Harness, written by Aleksey Shipilëv and the OpenJDK performance team — is the answer. It is the benchmarking framework in the JVM world. If someone shows you performance numbers that weren’t produced by JMH (or an equivalent — there really aren’t many), you are entitled to be skeptical.

1. Why Hand-Rolled Benchmarks Lie

Four specific traps kill most naive benchmarks:

Dead-Code Elimination (DCE). If the JIT can prove the result of your benchmark is unused, it will delete the computation. That empty loop that supposedly takes 3 ns per iteration? C2 elided the body; you’re timing loop overhead.

Constant Folding. If your inputs are compile-time constants, the JIT computes the result at compile time and hard-codes the answer. Math.sqrt(2.0) inlined into a loop with a literal is a table lookup, not a computation.

Insufficient warmup. For the first ~2 seconds, your code runs in the interpreter and C1. The C2-compiled version, which is what production actually runs, doesn’t exist yet. Measuring the first 100 ms of a loop measures a different program than the loop’s steady state.

Coordinated omission. In load-testing (not exactly microbenchmarking, but adjacent): if your test client stalls when the server stalls, you underestimate latency of the very requests that were slowest. Gil Tene’s original point. Solved in tools like wrk2 and hdrhistogram.

Read Shipilëv’s canonical short piece: shipilev.net/jvm/anatomy-quarks/27-compiler-blackholes/. If you internalize one performance-engineering essay, make it that one.

2. JMH Setup

JMH is not a library you drop on the classpath and use in your app — it’s a Maven/Gradle plugin that generates a separate benchmark JAR. Set up a submodule:

mvn archetype:generate \
  -DinteractiveMode=false \
  -DarchetypeGroupId=org.openjdk.jmh \
  -DarchetypeArtifactId=jmh-java-benchmark-archetype \
  -DgroupId=com.example \
  -DartifactId=benchmarks \
  -Dversion=1.0
cd benchmarks
mvn clean package
java -jar target/benchmarks.jar         # runs all benchmarks

Gradle users: id "me.champeau.jmh" version "0.7.2" in your plugins block does the equivalent.

3. Anatomy of a Correct JMH Benchmark

A minimal, correct example:

import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.infra.Blackhole;
import java.util.concurrent.TimeUnit;

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@State(Scope.Benchmark)
@Fork(value = 2, jvmArgsAppend = {"-Xms1g", "-Xmx1g"})
@Warmup(iterations = 5, time = 1)
@Measurement(iterations = 10, time = 1)
public class StringConcatBenchmark {

    @Param({"5", "50", "500"})
    int size;

    String[] tokens;

    @Setup(Level.Trial)
    public void setup() {
        tokens = new String[size];
        for (int i = 0; i < size; i++) tokens[i] = "tok-" + i;
    }

    @Benchmark
    public String plusOperator() {
        String out = "";
        for (String t : tokens) out = out + t;
        return out;   // returning consumes the result — defeats DCE
    }

    @Benchmark
    public String stringBuilder() {
        StringBuilder sb = new StringBuilder();
        for (String t : tokens) sb.append(t);
        return sb.toString();
    }

    @Benchmark
    public void plusOperatorBlackhole(Blackhole bh) {
        String out = "";
        for (String t : tokens) out = out + t;
        bh.consume(out);
    }
}

What every annotation is doing:

Annotation	Purpose
@BenchmarkMode	Throughput, AverageTime, SampleTime, SingleShotTime, All. Pick per question.
@OutputTimeUnit	Reporting units. Nanoseconds for tight loops, microseconds for typical, milliseconds for slow.
@State(Scope.Benchmark)	State shared across threads; Scope.Thread for per-thread; Scope.Group for producer-consumer style.
@Fork(2)	Fresh JVMs per benchmark. Two forks minimum — catches JVM-to-JVM variance.
@Warmup(5, time=1)	Five 1-second warmup iterations. C2 gets to compile.
@Measurement(10, time=1)	Ten 1-second measurement iterations. Averaged with std deviation.
@Param	Runs the benchmark once per value — gives you a matrix, not a single number.
@Setup(Level.Trial/Iteration/Invocation)	Setup timing: once per JMH trial / per iteration / per benchmark call.
@Benchmark	The measured method. Return the result or use Blackhole.consume.

4. Blackholes: How JMH Defeats DCE

A Blackhole is a JMH-provided sink that consumes values in a way the JIT cannot prove is unused. Rules:

1. Return the result from your @Benchmark if you produce one value. JMH consumes returns.

2. bh.consume(x) if you produce many values or the return doesn’t cover them.

3. Never write int junk = someComputation(); and expect it to preserve the work — C2 will delete it.

Example of DCE going wrong without a blackhole:

@Benchmark
public void wrongUnusedResult() {
    int sum = 0;
    for (int i = 0; i < 1000; i++) sum += i * i;
    // sum is unused — DCE erases the entire loop.
}

JMH will warn you about “benchmark method returned nothing” — heed the warning. The fix is bh.consume(sum) or return sum;.

Shipilëv has a newer compiler-integrated blackhole (-Djmh.blackhole.mode=COMPILER, experimental in recent JMH) that eliminates blackhole overhead itself. Worth knowing exists, not worth changing your defaults over.

5. Reading JMH Output

A JMH run ends with a summary table whose columns are Benchmark, (param), Mode, Cnt, Score, Error, Units. A single line looks like this:

StringConcatBenchmark.plusOperator     size=500   avgt   20   Score ± Error   ns/op

Interpret each column:

• Score is the average (or throughput, for Mode.Throughput).

• Error is the 99.9 % confidence interval half-width. If two benchmarks’ Score ± Error ranges overlap, you cannot claim one is faster than the other.

• Cnt = forks × measurement iterations. With @Fork(2) and @Measurement(10) you get 20.

• Units = ns/op for AverageTime, ops/s for Throughput, and so on.

The classic result you should expect from a run like the one above: at size=5, plusOperator and stringBuilder are within a few nanoseconds of each other — because javac (Java 9+) rewrites straight-line a + b + c into a StringBuilder.append chain via invokedynamic. At size=500, plusOperator inside the for loop explodes: each += copies the whole prefix, so total work is O(n²). StringBuilder stays O(n). By the time the size hits a few hundred elements the gap is typically two orders of magnitude, and the error bars stop overlapping long before that. This is why “just use +” is fine for constant-count concatenation but disastrous inside a loop.

6. Profiling Inside JMH

JMH ships with built-in profilers you can attach with -prof:

java -jar target/benchmarks.jar StringConcat -prof gc          # GC allocation rate per op
java -jar target/benchmarks.jar StringConcat -prof stack       # stack sampling profiler
java -jar target/benchmarks.jar StringConcat -prof perfnorm    # Linux perf counters (cycles, instructions, IPC)
java -jar target/benchmarks.jar StringConcat -prof async:output=flamegraph;dir=/tmp/flames   # Async Profiler integration

-prof gc alone is worth the price of admission. It tells you bytes allocated per operation — a number that predicts long-run GC pressure better than any wall-clock measurement. Two benchmarks with the same score but different alloc.norm (bytes/op) will behave very differently under production load.

7. Common JMH Pitfalls

Even with JMH, you can mess up. The frequent traps:

• Shared mutable state without @State. Two threads writing to a field concurrently invalidates the measurement.

• Level.Invocation setup on nanosecond benchmarks. The setup overhead swamps the measurement. Use Level.Iteration unless per-invocation state truly matters.

• Overly-large @Param grids. @Param({"1","10",..."1000000"}) on multiple parameters explodes into hundreds of runs. Start small, expand deliberately.

• Benchmarking on a laptop with turbo boost, thermal throttling, and a Slack notification during the run. Use a quiet machine, plug in the power supply, disable turbo if you can, and always run twice to check reproducibility.

• Not versioning your benchmark code. A benchmark’s result is only meaningful in context: JDK version, JMH version, hardware, @Fork args. Check the whole benchmark project into git next to the results.

8. When Not to Use JMH

JMH is a microbenchmark harness. For macro-level performance questions — “how does my whole HTTP service perform under load?” — use a proper load generator (wrk, wrk2, k6, Gatling, JMeter) that hits the real service and reports latency percentiles. JMH answers questions about pieces; load testing answers questions about wholes. Do not use JMH to “benchmark my Spring controller”; use k6 hitting the controller’s endpoint.

⚠️ What Most People Get Wrong

“I ran it in a loop 100 times and averaged the last 50 — that’s basically JMH.” No. Averaging doesn’t defeat DCE. It doesn’t defeat constant folding. It doesn’t fork a fresh JVM so a lucky bimorphic call site in one run doesn’t taint the next. It doesn’t tell you the confidence interval. Every one of these traps is silent — the benchmark still produces a number, it’s just the wrong one. JMH exists specifically to defeat these traps; treat it as non-optional.

Reading numbers without error bars. 1.234 ns/op sounds precise. 1.234 ± 0.892 ns/op tells you the number is actually somewhere between 0.34 and 2.13 — more than a 6× range. If someone shows you a comparison without error bars, you have not been shown a comparison. You’ve been shown a rumor.

Recap

• Hand-rolled System.nanoTime() benchmarks are almost always wrong: DCE, constant folding, no warmup, no fork isolation.

• JMH is the framework. Not optional. @Benchmark, @State, @Fork, @Warmup, @Measurement, @Param.

• Use Blackhole.consume(...) or return the value to defeat DCE.

• Read output as Score ± Error; overlapping ranges = no difference proven.

• Add -prof gc for allocation rate, -prof async for flame graphs.

• For whole-service performance, load-testing tools (wrk2, k6, Gatling), not JMH.

---

Return to README.md · Previous: 04_diagnostic_tools.md · Next: 06_common_production_pathologies.md