03 — Benchmark Hygiene¶

If your benchmark takes 5 minutes to write, it is wrong. Assume it is wrong until you have used JMH.

Most Java benchmarks on the internet are incorrect. Not “slightly off” — fundamentally wrong, often by a factor of 10 or 100, sometimes measuring the wrong thing entirely. This is not because their authors are careless. It is because the JVM is smart enough to defeat naive benchmarks. Dead code elimination, on-stack replacement, tiered compilation, escape analysis, and inlining all conspire to make System.nanoTime() in a loop a lie.

You will do a lot of performance work in this roadmap (Phase 06, plus benchmarks scattered across every phase). If your benchmarks are wrong, every conclusion you draw from them is wrong, and you build false intuitions that take years to unlearn. So: hygiene first, then benchmarks.

The Rule¶

JMH or nothing.

JMH (Java Microbenchmark Harness) is the OpenJDK team’s own benchmarking tool. It is the only correct way to write Java microbenchmarks. If you are not using JMH, you are not benchmarking; you are guessing loudly.

Do not write your own harness. Do not use System.nanoTime(). Do not trust System.currentTimeMillis(). Do not use a Spring @PostConstruct hack. Use JMH.

There is one narrow exception: end-to-end integration benchmarks (“how many requests per second does my full REST endpoint handle”) are done with tools like wrk, k6, or gatling. Those are not microbenchmarks. For anything measuring code smaller than an HTTP request, use JMH.

Why `System.nanoTime()` In A Loop Is A Lie¶

A quick tour of the ways your naive benchmark deceives you:

Dead code elimination. If you compute a value and never use it, the JIT deletes the computation. Your “benchmark” measures the loop overhead of an empty loop.
Constant folding. If your input is a literal, the JIT folds it at compile time. Your benchmark measures returning a constant.
On-stack replacement (OSR). The JIT can swap a running interpreter frame for compiled code mid-loop. Your first 10ms and your last 10ms are measuring different things.
Tiered compilation. C1 compiles fast and slow; C2 compiles slow and fast. Your benchmark starts in the interpreter, hits C1, then hits C2. Early samples are 10-100x slower than late samples. Average of the whole run is nonsense.
CPU thermal throttling. On a laptop, especially in Indian summer, the CPU slows down mid-run to protect itself. Your last iteration is honestly slower than your first because the silicon is hot.
GC pauses. A single Full GC in the middle of a run can add 100ms. If you didn’t warm up the heap, GC behavior early and late is different.

JMH handles all six. Rolling your own does not.

The Minimum Viable JMH Benchmark¶

Here is the shape of a correct JMH benchmark. Learn this pattern and use it every time.

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@Warmup(iterations = 5, time = 1)
@Measurement(iterations = 10, time = 1)
@Fork(value = 2, jvmArgs = {"-Xms1G", "-Xmx1G"})
@State(Scope.Benchmark)
public class ListGetBenchmark {

    @Param({"1000", "100000", "10000000"})
    private int size;

    private List<Integer> arrayList;
    private List<Integer> linkedList;
    private int[] indices;

    @Setup
    public void setup() {
        arrayList = new ArrayList<>();
        linkedList = new LinkedList<>();
        for (int i = 0; i < size; i++) {
            arrayList.add(i);
            linkedList.add(i);
        }
        Random r = new Random(42);
        indices = new int[1000];
        for (int i = 0; i < indices.length; i++) {
            indices[i] = r.nextInt(size);
        }
    }

    @Benchmark
    public int arrayListGet() {
        int sum = 0;
        for (int idx : indices) {
            sum += arrayList.get(idx);
        }
        return sum;   // returned so the JIT can't dead-code-eliminate it
    }

    @Benchmark
    public int linkedListGet() {
        int sum = 0;
        for (int idx : indices) {
            sum += linkedList.get(idx);
        }
        return sum;
    }
}

Every annotation matters. If you cannot explain in one sentence why each one is there, you are not ready to benchmark yet — read the JMH samples first.

The Non-Negotiables¶

Warmup matters¶

@Warmup(iterations = 5, time = 1) runs the benchmark for 5 seconds before measurement. This gets the JIT to fully compile the hot code. Without warmup you are measuring the interpreter and C1, not C2. Warmup is not optional.

Fork isolation matters¶

@Fork(value = 2) runs the benchmark in 2 separate JVM processes. This matters because the JVM’s optimization decisions are influenced by every method it has ever compiled in its life. If you run benchmarkA then benchmarkB in the same JVM, benchmarkB may inherit inlining decisions from benchmarkA and be faster (or slower) than it should be. Forking separates them.

value = 1 is the minimum honest fork count. value = 2 or 3 is better because you get variance across JVM instances. Never value = 0.

Reporting with error bars, not point estimates¶

JMH by default reports mean and 99.9% confidence interval. When you write up a result, always report:

Mean
Error (± x, from the confidence interval)
Number of iterations and forks
JDK version, JVM flags, GC used, hardware (CPU model, RAM)

A result of “ArrayList is faster” is not a result. “ArrayList get: 42.1 ± 0.8 ns/op vs LinkedList get: 180 ± 12 µs/op (JDK 21, G1GC, MacBook Pro M2, 16GB, 10 iterations, 2 forks)” is a result.

Compare like-for-like¶

When comparing two configurations, only one variable changes. Do not compare ArrayList on JDK 17 to LinkedList on JDK 21 and conclude anything about lists. Do not compare G1GC to ZGC on different heap sizes. Same JDK, same GC, same heap, same hardware, same background load. If you cannot control a variable, note it in the writeup.

Publish the raw numbers¶

When you post benchmark results (blog, LinkedIn, whatever — see 05_teach_to_learn.md), include the raw JMH output file, not just your chart. This lets people spot mistakes. It also builds trust. “Trust me, ArrayList is 100x faster” is worth nothing. Raw JMH output attached to a blog post is worth a lot.

Common Anti-Patterns You Will Be Tempted By¶

The blackhole trap. Forgetting to consume your benchmark’s output. If you don’t return the value or feed it to a Blackhole, the JIT deletes your work. Symptom: your benchmark reports 0.3 ns/op no matter what you do. Cause: you are measuring nothing.
The setup-in-benchmark trap. Putting expensive setup inside the @Benchmark method. Setup goes in @Setup, not inside the measured code.
The shared-state trap. Using @State(Scope.Benchmark) when you should use @State(Scope.Thread). Symptom: threaded benchmarks give wildly noisy results because threads are contending on state that should be per-thread.
The GC-during-warmup trap. Not sizing your heap. Symptom: variance is huge because occasional Full GCs happen mid-measurement. Fix: @Fork(jvmArgs = {"-Xms1G", "-Xmx1G"}) and pick a size that keeps you off the GC edge.
The single-run confidence trap. Running JMH once and treating the number as truth. Run at least 2 forks. If results across forks disagree, your benchmark is unstable and you need to fix it before publishing.

Required Reading¶

These are not optional. Watch or read them before you publish a single benchmark:

Aleksey Shipilëv, “Java Performance Puzzlers” — YouTube. Any of his talks. He is the person you go to on JMH. Search for talks at Devoxx, JavaZone, JVMLS.
Aleksey Shipilëv, “Nanotrusting the Nanotime” — blog post at shipilev.net. The definitive explanation of why timing is hard.
JMH samples — https://github.com/openjdk/jmh/tree/master/jmh-samples — read JMHSample_01 through JMHSample_15. There are 38 samples total; the first 15 cover everything you’ll encounter in Phase 06.
Brendan Gregg, “Systems Performance” — Chapter 1 (methodology). Not Java-specific but the discipline is universal.

Budget: 4-6 hours of watching + reading before your first serious benchmark. This is the highest-ROI reading in the entire roadmap.