03 — Benchmark Hygiene

Every C engineer eventually publishes a benchmark. Most of them are wrong. This is the file that keeps yours from being one of them. If your portfolio shows a benchmark, an experienced reader will check hygiene BEFORE they check the number — if the hygiene is bad, the number is discarded, and so are you.

How to measure honestly

The minimum viable protocol:

  1. Build in Release mode with the same flags you’d ship: -O2 -DNDEBUG at minimum, -O3 -march=native when appropriate. Never publish a -O0 benchmark unless the point of the benchmark IS the effect of optimization levels.

  2. Warm up. Run the benchmark 3–5 times before recording. First-run numbers include page-fault overhead, cold cache, JIT settling in the runtime, and CPU frequency ramp-up. They lie.

  3. Repeat and report variance. Minimum 10 runs. Report median and either standard deviation or interquartile range. A single number without variance is a lie by omission.

  4. Pin the CPU and disable turbo/thermal-throttling where you can. On Linux: taskset -c 3, cpupower frequency-set --governor performance, disable Intel Turbo Boost if reproducibility matters more than peak. On Mac, note the limitation in the report — you cannot pin cleanly on Apple Silicon.

  5. Use monotonic clocks. clock_gettime(CLOCK_MONOTONIC) in C. Never time() for sub-second work. Never gettimeofday() for anything new.

  6. Time CPU, not wall, when you mean CPU — and vice versa. CLOCK_PROCESS_CPUTIME_ID measures CPU. If your benchmark makes system calls or sleeps, wall clock lies about CPU cost.

  7. Use perf stat (Linux) for hardware counters when the wall-clock delta is small — cache misses, branch mispredictions, instructions per cycle. These tell you why, not just how much.

The 7 sins of amateur C benchmarking

  1. Dead-code elimination. You wrote int x = compute(); } and the compiler noticed x is unused and deleted compute() entirely. Your “benchmark” measures nothing. Fix: consume the result (asm volatile("" :: "r"(x)), or volatile sink, or print it).

  2. Ignoring cache effects. You loop over 10 KB of data 1M times and get a great number. Real workloads touch 100 MB and get destroyed by cache misses. Fix: benchmark at multiple sizes (L1-fit, L2-fit, LLC-fit, RAM-only) and publish all four.

  3. Running once. One number is not a measurement, it is an anecdote. Fix: minimum 10 runs, report median + variance.

  4. Comparing debug vs release builds. You built your version at -O0 and compared it to a library built at -O3. You concluded your version is 50x slower. It might be 1.2x slower. Fix: identical flags for both, or note the difference prominently.

  5. Forgetting -O2. Same sin as (4) but self-inflicted. Publishing a -O0 benchmark as if it says something about production code is a beginner tell.

  6. Timing wall vs CPU incorrectly. Sleeping 1s inside your function makes wall time useless. Doing heavy I/O makes CPU time useless. Fix: know what you’re measuring and pick the right clock.

  7. Not repeating on cold cache. You ran the same array 20 times; run 2 through 20 all hit L1. Real hot-path code sees the array once. Fix: if you want to measure cold performance, flush the cache between runs or use fresh data.

Tools worth learning

Tool

What it’s for

When to use

hyperfine

Wall-clock benchmarking of full programs; handles warmup, repeats, stats automatically

Any time you have a full binary to time

perf stat (Linux)

Hardware counters — cycles, instructions, cache misses, branch misses

When you need to explain WHY something is slow

perf record + perf report

Statistical profiler

Finding hot functions in a real workload

valgrind --tool=callgrind

Deterministic cycle counting (simulated)

When you need reproducibility over accuracy

google/benchmark

Micro-benchmark framework (C++, usable from C wrappers)

Micro-benchmarks of specific functions

bpftrace / bcc

Kernel and library tracing

When you suspect the problem is below your code

Note the tools you don’t need on day one: no flamegraphs, no VTune, no Intel Advisor. Start with hyperfine and perf stat. Add complexity only when the simple tools stop being enough.

Criterion-style comparison mode as inspiration

Rust’s Criterion.rs has one feature worth stealing conceptually: it stores the previous run’s numbers and prints the percentage change when you run again. You can approximate this in C:

  • Every benchmark run appends to a results.jsonl or results.csv.

  • A tiny script (Python is fine) diffs the last two runs and prints “+3.2% (regression)” or “-8.1% (improvement)”.

  • Commit results.csv to the repo. Now your git history shows performance history.

This is what turns a benchmark into a regression fence rather than a one-time marketing exercise.

The benchmark report template

Every benchmark you publish uses this exact structure. Deviation invites skepticism.

# Benchmark: <one-line-title>

## Environment
- CPU: <model, cores, base freq, turbo>
- RAM: <GB, DDR type, speed>
- OS: <name + kernel version>
- Compiler: <gcc/clang + version>
- Flags: <exact flags used>
- Governor / turbo: <performance / turbo off / etc.>
- Repeats: <N>
- Warmup runs: <M>

## What is being measured
<One paragraph: the operation, the input, what "one run" means.>

## Baseline
<What are we comparing against? Another implementation, a library, a previous version.>

## Results

| Variant | Median | p50 | p99 | Std dev | vs baseline |
|---|---|---|---|---|---|
| … | … | … | … | … | … |

## Hardware counters (if relevant)
- Cycles, instructions, IPC, cache misses, branch misses

## Caveats
- What could still be wrong about this measurement
- What real-world scenario this does or does not model

## Reproducing
- Exact command line
- Link to source at commit hash

If your benchmark blog post does not have all these sections, do not publish it. A benchmark without caveats is not a benchmark; it is a boast.

The final rule

Assume every number you publish will be run again by a stranger who wants to prove you wrong. Design the benchmark so that when they do, they arrive at your number. If they can’t, your benchmark was propaganda.

Return to README.md · Next: 04_daily_practice.md