03 — Benchmark Hygiene

C++ is a performance language. If you cannot measure honestly, you cannot claim performance. This file is the operating manual for measurement on your MacBook (Apple Silicon).

Every benchmark you publish — in a README, a blog post, a resume bullet, an study — must satisfy the checklist at the bottom of this file. No exceptions. Sloppy benchmarks are worse than no benchmarks: they destroy your credibility with the exact senior engineers you want to be evaluated by.

The 8 rules

Rule 1: Release, not Debug

Measure only Release builds. Debug builds have zero-cost abstractions turned off, iterator debugging turned on, and inlining suppressed — they are 5–50x slower than Release for the same code. A Debug-build benchmark is not just wrong, it is misleading in a direction that flatters your competitors.

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j

Verify: strings build/your_binary | grep -i debug should return nothing informative. Or check -O flag: your build log must show -O2 or -O3.

Rule 2: Compiler flags disclosed

Every benchmark discloses: compiler, version, flags. E.g.:

Apple clang 17.0.0 (arm64-apple-darwin24), -O3 -std=c++20 -DNDEBUG -march=native

Without this line, the number is meaningless. -O2 vs -O3 can be 30% either way. -march=native unlocks NEON instructions on M-series and can be 2–4x on vectorisable loops.

Rule 3: Hardware disclosed

MacBook Pro 14" M2 Pro, 10-core (6P+4E), 16 GB unified memory, macOS 14.6.

Benchmark numbers without hardware context are trivia. An M2 Pro vs an M4 Max on the same code can be 2x. An intel MacBook is a different universe.

Rule 4: Baseline named

Never write “fast” or “efficient” without a named baseline. Correct forms:

  • “1.7x faster than std::sort on vector<int> size 1e6”

  • “2.3x slower than Eigen 3.4 MatrixXf::operator* on 512×512”

  • “Matches OpenBLAS sgemm within 8% on M2 Pro for N=1024”

A benchmark without a baseline is a lie by omission. It says “look at how fast I am” and hides “faster than what?”

Rule 5: Warm-up runs

Run the code once (or 3x for cache-heavy work) before the measurement loop. First runs suffer from cold cache, cold TLB, dyld resolving symbols, and — on newer macOS — the M-series core-selector taking a few hundred microseconds to promote your thread to a performance core.

// Warm-up
for (int i = 0; i < 3; ++i) { workload(); }

// Measure
auto t0 = std::chrono::steady_clock::now();
for (int i = 0; i < N; ++i) { workload(); }
auto t1 = std::chrono::steady_clock::now();

Rule 6: Median of N=10, not mean

Mean is dragged around by outliers (a GC pause in the OS, a Spotlight indexing burst, a notification from Slack). Median is robust.

std::array<double, 10> times{};
for (auto& t : times) t = measure_one();
std::sort(times.begin(), times.end());
double median = times[times.size() / 2];

Report: median, min, and standard deviation. If stddev / median > 15%, your measurement is noisy — fix the setup before publishing.

Rule 7: Disable CPU frequency scaling (or acknowledge it)

On Apple Silicon, macOS aggressively down-clocks efficiency cores and can also throttle performance cores when the chassis heats up. Before a benchmark run:

  1. Plug in the charger. On battery, macOS caps performance more aggressively.

  2. Disable low power mode:

    sudo pmset -a lowpowermode 0

  3. Close everything. Chrome, Slack, Docker, Zoom. Especially Docker — the VM steals cycles unpredictably.

  4. Cool state. Wait 60 seconds after a heavy build before running the benchmark. Thermal throttling on a fanless MacBook Air will destroy your numbers.

  5. Pin to a performance core if you can — on macOS this is imperfect, but QOS_CLASS_USER_INTERACTIVE biases the scheduler toward P-cores:

    pthread_set_qos_class_self_np(QOS_CLASS_USER_INTERACTIVE, 0);

If you cannot fully disable scaling (macOS does not expose full control the way Linux does with cpupower), acknowledge it in the README and run 3 separate sessions on different days — if medians agree within 5%, you are stable enough to publish.

Rule 8: Prevent compiler dead-code elimination

This is the classic C++ benchmark trap. If the compiler can prove your workload’s output is unused, it may delete the entire loop. You then measure the empty loop and publish 900 GB/s.

Use benchmark::DoNotOptimize (from Google Benchmark) or a hand-rolled equivalent:

template <typename T>
inline void do_not_optimize(T const& v) {
    asm volatile("" : : "r,m"(v) : "memory");
}

And sink the result:

auto result = workload();
do_not_optimize(result);

Sanity check: if your benchmark clocks in at < 1 ns/op for anything non-trivial, the compiler ate your loop. Investigate.

Preferred tools

  • Google Benchmark — default choice. Handles warm-up, iteration count, and DoNotOptimize for you. Add it via CMake FetchContent or vcpkg.

  • std::chrono::steady_clock — for one-off microbenchmarks where google/benchmark is overkill. Never use std::chrono::system_clock (wall clock, can go backwards).

  • hyperfine — for full-binary benchmarks (“how long does ./myserver process input.txt take end-to-end?”). Handles warmups, statistical output, and comparisons. brew install hyperfine.

  • Instruments.app — for profiling, not benchmarking. Use “Time Profiler” and “System Trace” templates. Free with Xcode.

The benchmark README template

Copy this block into every project’s README.md that publishes numbers. Fill in the blanks. If you cannot fill a field, do not publish the number.

## Benchmarks

### Setup
- Hardware: MacBook Pro 14" M2 Pro, 10-core (6P+4E), 16 GB, macOS 14.6
- Compiler: Apple clang 17.0.0 (arm64-apple-darwin24)
- Flags:    -O3 -std=c++20 -DNDEBUG -march=native
- Build:    cmake --build build --config Release
- Baseline: <named library and version>
- Method:   Google Benchmark, min 10 iters, warmup 3, wall-clock median
- Power:    plugged in, low-power mode OFF, Docker off, Chrome closed

### Numbers
| Workload            | This code (median) | Baseline (median) | Ratio | Stddev/med |
|---------------------|-------------------:|------------------:|------:|-----------:|
| vec-add 1e6 f32     |            1.2 ms  |          1.1 ms   | 1.09x |       3.2% |
| matmul 512×512 f32  |          14.7 ms  |         12.0 ms   | 1.23x |       6.1% |

### Caveats
- macOS does not expose full CPU-freq pinning; numbers vary ~5% across sessions.
- All measurements on P-cores via QOS_CLASS_USER_INTERACTIVE.
- Results not validated on Linux/x86_64.

The 30-second pre-publish checklist

Before any benchmark number leaves this project’s directory:

  • Release build only?

  • Compiler + flags + hardware + macOS version stated?

  • Named baseline (library + version)?

  • N ≥ 10, median (not mean)?

  • Stddev / median ≤ 15%?

  • Warm-up runs done?

  • Charger plugged in, low-power off, Docker/Chrome off?

  • DoNotOptimize used on the result?

  • Caveats section written honestly?

All nine boxes ticked — publish. Any unchecked — fix before publishing.