Benchmark Hygiene — How to Measure Honestly¶
Dishonest measurement is worse than no measurement. It gives you false confidence in a direction that’s wrong.
The Core Problem¶
ML practitioners, especially self-taught ones, are uniquely vulnerable to self-deception in measurement. Unlike software engineering, where “does it compile and pass tests” is a binary check, ML performance exists on a continuous scale with enormous room for self-serving interpretation.
You can make almost any model look good if you:
Choose the right metric
Pick the right dataset split
Run enough seeds and report the best one
Compare against a weak baseline
Peek at the test set while developing
None of these feel like fraud in the moment. They feel like “finding the result.” This section exists to eliminate that drift.
The 7 Benchmark Anti-Patterns¶
Anti-Pattern 1: Test Set Peeking¶
What it looks like: Using test set performance to guide architectural or hyperparameter decisions.
Why it’s wrong: You’ve now trained on the test set implicitly. Any reported test performance is optimistic.
The fix: The test set is touched exactly once — at the very end, when you’ve finished all development. Not before.
Anti-Pattern 2: Metric Shopping¶
What it looks like: Running 5 metrics, reporting only the one that looks best.
Why it’s wrong: You’re not reporting performance; you’re reporting the best-case scenario.
The fix: Decide which metrics you’ll report before you run the experiment. Report all of them.
Anti-Pattern 3: Ignoring Confidence Intervals¶
What it looks like: “My model achieves 87.3% accuracy.”
Why it’s wrong: 87.3% on one run means almost nothing. Is this 87.3 ± 0.2% or 87.3 ± 4.1%?
The fix: Report mean ± std over at least 3 runs with different seeds.
Anti-Pattern 4: Comparing Across Datasets¶
What it looks like: Comparing your model trained on one preprocessing pipeline to a baseline trained on another.
Why it’s wrong: You may be measuring data preparation quality, not model quality.
The fix: Ensure identical data splits, preprocessing, and evaluation conditions before any comparison.
Anti-Pattern 5: Cherry-Picking Seeds¶
What it looks like: Running 10 seeds, reporting the best 3.
Why it’s wrong: This is selection bias. You’ve cherry-picked the lucky initializations.
The fix: Fix the seed set in advance. Report all runs. If variance is high, that’s a finding worth reporting.
Anti-Pattern 6: Moving Baselines¶
What it looks like: Your baseline is whatever makes your approach look best.
Why it’s wrong: A strong method should beat a strong baseline, not a weak one.
The fix: Use the strongest publicly available baseline for your task. If you beat a weak baseline, state clearly that you chose it because of compute constraints, not because it’s the fair comparison.
Anti-Pattern 7: Incomplete Reporting¶
What it looks like: “My model converges faster.” (No numbers. No comparison method. No definition of convergence.)
Why it’s wrong: Unfalsifiable claims are scientifically worthless.
The fix: Every claim must have a number. “Converges to 85% validation accuracy in 40% fewer epochs than the Adam baseline, measured as epochs to threshold, mean over 5 seeds.”
The Honest Measurement Protocol¶
Apply this to every experiment worth documenting:
HONEST MEASUREMENT CHECKLIST
==============================
Before running the experiment:
[ ] Metrics chosen and locked: ________________________
[ ] Baseline defined: ________________________
[ ] Data splits fixed (and not touched since splitting): Y / N
[ ] Seeds: I will use seeds [____], [____], [____]
[ ] What I'm measuring is actually what I care about: Y / N
(If not, what's the gap? ________________________)
During the experiment:
[ ] All runs logged (not just the good ones)
[ ] Validation set used for development; test set untouched
After the experiment:
[ ] Report: mean ± std over ≥ 3 runs
[ ] ALL metrics reported (not just the favorable ones)
[ ] Test set touched exactly once: Y / N
[ ] Comparison to baseline done under identical conditions: Y / N
How to Interpret Learning Curves¶
Learning curves are one of the most diagnostic tools available. Read them before drawing conclusions from a single number.
Pattern |
What It Means |
What To Do |
|---|---|---|
Training loss ↓, Validation loss ↓ together |
Model is learning, not overfit |
Continue. Increase capacity if both plateau high. |
Training loss ↓, Validation loss plateau/↑ |
Overfitting |
Regularization, more data, reduce capacity |
Both losses plateau high early |
Underfitting or learning rate too low |
Increase capacity, check LR, check gradients |
Both losses oscillate without convergence |
Learning rate too high or unstable optimization |
Reduce LR, try gradient clipping |
Validation loss → NaN |
Numerical instability |
Check for exploding gradients, normalize inputs |
Training loss ≈ Validation loss but both poor |
Underfitting — model too simple |
Increase capacity or improve features |
Large gap between train and val from epoch 1 |
Data leakage or distribution mismatch |
Audit the data pipeline |
How to Know If You’re Actually Learning¶
This is the question that discipline is supposed to answer. Vanity metrics (hours studied, pages read, courses completed) tell you nothing useful.
The real test is backward-looking capability delta.
At the start of each sprint, answer:
“What can I do today that I couldn’t do 2 weeks ago?”
If you can’t answer that question with a specific, demonstrable capability — you’re completing exercises, not building skills.
Weekly Self-Assessment¶
Run this on Friday alongside the lab notebook weekly review.
WEEKLY SELF-ASSESSMENT
======================
Date: ____
CAPABILITY DELTA (2 weeks ago vs. today):
"I can now [specific task] that I could not do 2 weeks ago."
Answer: ________________________________________________________________
(If blank: this is a red flag. What happened this sprint?)
BENCHMARK HYGIENE CHECK:
[ ] All experiments this week used pre-committed metrics
[ ] No test set peeking
[ ] Results reported with variance (not just point estimates)
[ ] Comparisons done under fair conditions
ONE HONEST ASSESSMENT:
Where am I being sloppy with measurement?
________________________________________________________________
CALIBRATION CHECK:
Of my predictions this week, approximately ____% were correct.
Is my calibration improving? Y / N / Not enough data
A Note on “Good Enough” vs. “Rigorous”¶
You are not writing papers. You don’t need to run 50 seeds. You don’t need to beat SOTA on ImageNet.
What you need is honest measurement that builds accurate intuition. Three seeds and pre-committed metrics is achievable and sufficient for a learning project. The discipline is not about perfection — it’s about not lying to yourself.
If you can’t reproduce your own results from 3 weeks ago, your measurement discipline is insufficient. Fix it.