03 — Benchmark Hygiene as Identity¶
A benchmark you cannot reproduce is a lie you told yourself.
Why this is an identity file, not a technique file¶
In 13 months, you are going to publish numbers. On r/LocalLLaMA, on a blog, in a PR, in an study loop. The first time someone senior spot-checks one of those numbers and finds it doesn’t reproduce — or worse, finds it was measured with time.time() around a CUDA op — you lose credibility that takes years to rebuild.
Benchmarking is not a phase. It’s not something you’ll “get better at later.” It is the first professional habit of an inference engineer, and it starts in sprint 1 with the smallest measurement you take.
Every benchmark you publish should carry your name like a signature. Someone should be able to clone the repo, run the script, and get your number within noise. If they can’t, you didn’t publish a benchmark — you published a claim.
The seven rules¶
1. Warmups (≥3 iterations, discarded)¶
GPUs, drivers, allocators, kernel autotune caches, and JIT compilers all have first-run costs. Your first measurement is always wrong.
# WRONG
start = time.time()
for _ in range(10):
run()
print((time.time() - start) / 10)
# RIGHT
for _ in range(3): # warmup, discard
run()
torch.cuda.synchronize()
# ... then time ...
For Triton or torch.compile: warmup ≥5 iterations because autotune runs on first call.
2. CUDA events for GPU timing (never time.time())¶
time.time() measures wall clock, which is fine for CPU code but wrong for GPU code because kernel launches are asynchronous.
# The ONLY correct way to time a GPU op in PyTorch
start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)
start.record()
result = my_gpu_op(x)
end.record()
torch.cuda.synchronize() # this is mandatory
elapsed_ms = start.elapsed_time(end)
For multi-op timing, use torch.profiler or Nsight Systems. Never trust a %%timeit cell without torch.cuda.synchronize() inside it.
Exception: for end-to-end serving benchmarks (HTTP-in, HTTP-out), wall clock is correct because that’s the metric that matters. But for kernel-level, CUDA events.
3. Lock clocks (defeat thermal / power-state variance)¶
Without this, your “25% speedup” might just be the GPU boosting because the room got cooler.
# List available clocks
nvidia-smi -q -d SUPPORTED_CLOCKS
# Lock GPU clock (example: 1695 MHz on RTX 3090)
sudo nvidia-smi -lgc 1695
# Lock memory clock too
sudo nvidia-smi -lmc 9501
# Unlock afterwards
sudo nvidia-smi -rgc
sudo nvidia-smi -rmc
Rule: lock clocks for any benchmark you’re going to publish or compare across days. Skip this for exploratory work, but never for a number that ends up in a writeup.
4. Pin the environment in the writeup¶
Every published benchmark must include:
Hardware: NVIDIA RTX 3090 (Ampere, GA102), 24GB, PCIe 4.0 x16
Driver: 550.107.02
CUDA: 12.4
PyTorch: 2.4.1+cu124
vLLM: 0.6.3.post1 (commit abc123)
OS: Ubuntu 22.04, kernel 6.5.0
Clock lock: 1695 MHz GPU / 9501 MHz mem
Power mode: nvidia-smi -pm 1, power limit 350W
Yes, all of it. Yes, every time. Copy-paste from a env_snapshot.sh you commit to the repo:
#!/bin/bash
# env_snapshot.sh — dump to a file, include in benchmark writeups
nvidia-smi --query-gpu=name,driver_version,pstate,clocks.gr,clocks.mem,power.limit --format=csv
nvcc --version
python -c "import torch; print(torch.__version__, torch.version.cuda)"
pip freeze | grep -E "vllm|sglang|flash-attn|triton|transformers"
uname -a
5. Always p50/p95/p99 — never means¶
A mean latency for a serving benchmark is a lie. LLM latency distributions are heavy-tailed. Your users care about p99. Your SLO is on p99. Report:
latency (ms) p50 p90 p95 p99 max
TTFT 48 72 91 143 412
ITL 21 28 34 47 181
E2E (256 out) 5.4s 6.9s 7.4s 9.1s 14.2s
For kernel-level benchmarks where variance is small (locked clocks, no other tenants), report median + stdev across N≥30 runs. Mean is acceptable only when you’ve shown stdev/median < 2%.
Goodput > throughput. If a serving benchmark reports “throughput = 2400 tok/s” without saying what fraction met the SLO, it’s a marketing number. From the DistServe paper on: report goodput at the SLO threshold. That’s the honest metric.
6. Commit the benchmark script (before you commit the number)¶
Rule: the writeup and the script land in the same PR. Not “I’ll clean it up later.” Not “the script is on my other machine.” If it’s not in git, the number doesn’t exist.
Structure:
repo/
├── bench/
│ ├── env_snapshot.sh
│ ├── fa2_kernel.py # the script
│ ├── fa2_kernel_configs.yaml # sweep config
│ └── results/
│ ├── 2026-08-08_3090.json
│ └── 2026-08-08_3090.md # the writeup
Writeup lives next to results. Results are JSON so future-you can re-plot them. Script has a --dry-run mode that prints the config without running, so reviewers can verify what you claim to have measured.
7. Realistic workloads (or say what you’re doing)¶
For kernel benchmarks: sweep meaningful dimensions. seq ∈ {128, 512, 1024, 2048, 4096, 8192}; batch ∈ {1, 4, 16, 64}; dtype ∈ {fp16, bf16}. Don’t cherry-pick one shape.
For serving benchmarks: use realistic arrival and length distributions.
Arrival process: Poisson at target QPS, not uniform.
vllm bench serve --request-rate <qps>does this correctly.Length distribution: ShareGPT-style (bimodal, long tail on outputs), not fixed 512-in/128-out.
Concurrency: sweep, don’t fix. Report the throughput-latency curve, not one point.
Reference: vllm bench serve is the community convention. Use it or match its interface. Deviating without documenting why is a red flag to any reviewer.
The pre-publish checklist¶
Before any number leaves your machine (blog, Reddit post, PR, tweet, resume), tick every box:
[ ] Warmup iterations: ≥3 (or ≥5 for autotuned code)
[ ] Timing: CUDA events (kernel) or wall clock at request boundary (serving)
[ ] Clocks locked (nvidia-smi -lgc / -lmc), lock values in writeup
[ ] env_snapshot.sh output pasted into writeup
[ ] p50 / p95 / p99 reported (serving) OR median + stdev (kernel)
[ ] Benchmark script committed at commit hash [___________]
[ ] Results JSON committed at commit hash [___________]
[ ] Workload: realistic (Poisson + ShareGPT for serving; sweep for kernel)
[ ] Comparison baseline: named, version-pinned, commit-pinned
[ ] Ran on isolated GPU (no other tenants; nvidia-smi shows only my PID)
[ ] Re-ran once on a fresh boot and got within noise (or noted the delta)
[ ] For serving: SLO stated explicitly (e.g. p99 TTFT < 200ms); goodput reported
[ ] Someone else could clone repo, `bash bench/run.sh`, and get my number
Print this. Tape it above your monitor. Tick every box, every time.
Common cheats you’re going to be tempted to do (don’t)¶
“I’ll skip warmups this one time, the number won’t change much.” It will. The first iteration is often 3-10x slower.
“I forgot to lock clocks but the number matches roughly.” Roughly is not a benchmark. Re-run.
“I’ll report mean because p99 looks bad.” That’s exactly the point of p99.
“I’ll compare vLLM at commit X to TGI at commit Y-from-last-year.” Pin both to the same week or don’t compare.
“I’ll run on my laptop while I watch a stream.” Background load = invalid benchmark. Full stop.
“I benchmarked at batch=1 because it was easy.” Batch=1 is a diagnostic, not a benchmark. Serving happens at batch >> 1.
“I’ll drop the outlier at run 7, it’s clearly noise.” No. Investigate it. Real outliers explain the tail latency that will kill you in production.
The vLLM bench convention (learn it, use it)¶
vllm bench serve is the reference workload runner in the ecosystem. Its conventions:
vllm bench serve \
--backend vllm \
--model meta-llama/Llama-3-8B \
--dataset-name sharegpt \
--dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
--num-prompts 1000 \
--request-rate 8.0 \
--seed 42
Output reports: request throughput (req/s), output token throughput (tok/s), TTFT (p50/p95/p99), ITL (p50/p95/p99), E2E latency, and goodput. Read the source of benchmarks/benchmark_serving.py in vLLM once. Understand what every number means. Then any serving benchmark you publish either uses this or matches this interface — so anyone can reproduce.
SGLang has an equivalent (python -m sglang.bench_serving). Same rules.
The identity claim¶
Eventually — sooner than you think — someone will forward one of your benchmarks internally at Neural Magic or LMSYS or Fireworks with a note like “this one actually reproduces.”
That sentence is your career. It’s earned one number at a time, one locked clock at a time, one committed script at a time. Every shortcut you take now is a discount on that future compliment.
Be the person whose numbers reproduce.