How to Read ML Papers

Reading a paper is not the same as understanding it, and understanding it is not the same as being able to critique it. Most engineers who say “I read that paper” mean they absorbed the abstract and glanced at Figure 1. This document describes how to actually read a paper — which is a distinct, learnable skill that separates engineers who track the field from engineers who move the field.

The single prerequisite: you must read with a question in mind. Papers are written by people who already had a question. If you approach one without your own question, you are reading someone else’s narrative and absorbing their framing uncritically. Every paper reading session should start with: “What do I want to know from this paper, and what will I do with that knowledge?”

The Keshav 3-Pass Method

This is the established standard, first formalized by S. Keshav in “How to Read a Paper” (2007) and widely adopted in the ML community. It applies directly to ML papers with minor adaptation.

Pass 1: The Skim (15–20 minutes)

Goal: Decide if the paper deserves Pass 2.

  • Read: title, abstract, section headings, conclusion, figures with captions

  • Ignore: equations, proofs, experimental details

  • Answer these 4 questions: What category is this (new architecture / new training method / benchmark / analysis)? What context is it placed in (what field, what prior work)? What correctness assumption am I making? What contributions do the authors claim?

  • Output: KEEP or DISCARD decision. If DISCARD, note why — that judgment is itself data.

Pass 2: The Structure Read (1–2 hours)

Goal: Understand the core argument. You should be able to summarize the paper to someone else afterward.

  • Read everything except proofs. Mark unclear sections — do not stop on them.

  • Annotate figures. In ML papers, the key insight is almost always in 1–2 figures. If you can reproduce the figures mentally, you understand the paper.

  • Verify that the abstract’s claims map to the experiments. Do the results actually demonstrate what the abstract says? This single check catches ~40% of overclaimed papers.

  • Output: 3–5 bullet notes. The claim, the mechanism, the evidence, the strongest weakness you see.

Pass 3: The Deep Read (3–5 hours)

Goal: Be able to reconstruct the paper’s contribution from first principles.

  • Read every equation. Derive the key ones on paper.

  • Critically examine every experimental choice: what baselines were used, what datasets, what hyperparameters. Ask: who chose these and why? Are there suspiciously absent baselines?

  • Attempt the “3-hour reproduction test” (see below).

  • Output: Full annotation. Your opinion of the paper. What you would test if you had unlimited compute.

Most papers deserve Pass 1 only. Pass 2 for papers that are directly relevant to your work. Pass 3 for foundational papers in your area of depth — roughly 4–6 per year.

The 5 Questions to Ask of Every ML Paper

These are not optional. Ask them explicitly, in writing, for any paper you read at Pass 2 or deeper.

Q1: What problem does this paper actually solve?

Not what the authors say they solve — what measurable gap in capability or efficiency does this paper address? If the problem is vaguely defined, the paper’s contribution may be equally vague. Red flag: “we propose a novel framework for…” — “framework” is a word that hides weak contributions.

Q2: What prior work does this supersede, and by how much?

Every ML paper claims to beat prior work. Find the actual delta. If a paper claims “state-of-the-art” on a benchmark: (a) is this benchmark saturated? (b) is the improvement within the variance of the evaluation? (c) is the baseline recent? A 0.3 BLEU improvement over a 2-year-old baseline is not state of the art.

Q3: What is the core mechanism?

Strip away the implementation details. What is the single algorithmic or architectural idea that makes this work? If you cannot state it in 2 sentences, you do not yet understand the paper. Example for Flash Attention 2: “Attention computation is tiled to operate within SRAM rather than reading/writing HBM, reducing memory bandwidth from O(N²) to O(N).”

Q4: What evidence is provided, and is it sufficient?

Examine: number of seeds, statistical significance tests (or absence), evaluation set leakage, cherry-picked examples, ablation completeness. The most common failure mode in ML papers: ablations that only remove components that don’t hurt, never components that would reveal the true driver of performance.

Q5: Where does this method break?

Every method has a failure mode the paper does not prominently advertise. Scan the appendix, the limitations section (if it exists), and the negative results. If there is no limitations section — that is itself evidence the authors don’t fully understand their method. Example: Mamba/SSM has known weaknesses in input copying and in-context retrieval tasks that are buried in follow-up work, not the original paper.

How to Critically Evaluate Ablations

Ablation studies are where most of the truth in ML papers lives — and where most of the spin occurs. A rigorous ablation removes exactly one component at a time and measures the performance impact. An inadequate ablation removes components that are either irrelevant or always beneficial.

Red flags in ablations:

  • Only ablating architectural choices, not training choices (learning rate, batch size, data augmentation affect results as much as architecture)

  • Showing only the best checkpoint, not training curves

  • Ablating on a different dataset than the main results

  • No variance estimates across seeds

  • “We ablated X and found it didn’t help” — without the actual numbers

What a good ablation looks like: Each component is removed independently. The results show which components drive the performance. You can identify the Pareto-optimal configuration: the simplest model that gets 90% of the performance. If removing the paper’s key contribution barely moves the number, the paper has a problem.

How to Read Results Sections Without Being Fooled

The selective reporting problem: Authors choose which benchmarks to report, which baselines to compare against, and which metrics to highlight. This is not fraud — it is human nature. Your job is to compensate for it.

Checklist for any Results section:

  • Is the evaluation set public and standard? Or custom?

  • Are the baselines the strongest available, or convenient ones?

  • Are results reported with variance (stddev, confidence intervals)?

  • Is there a “best single run” vs. “median of N runs” distinction?

  • Does the paper report failure cases or only successes?

  • For LLM papers: is the evaluation on a benchmark that has been leaked into training data? (This is widespread and material.)

The Goodhart’s Law trap: When a metric becomes the target, it ceases to be a valid measure. Perplexity is a useful training metric and a weak proxy for downstream task performance. BLEU is a useful proxy for translation quality and an unreliable measure of semantic correctness. MMLU is a useful benchmark and a saturating one (models score 90%+ now). Know what each metric actually measures before trusting improvement claims on it.

The “3-Hour Reproduction Test”

The Feynman test applied to ML: if you cannot implement the core mechanism, you do not understand it.

The test: given a paper, can you implement the essential algorithm (not the full system) in 3 hours using only PyTorch/NumPy and the paper itself?

What counts as passing:

  • You implement the core forward pass / training loop

  • Your implementation produces numbers in the right order of magnitude

  • You can explain every line of your code in terms of the paper’s equations

What does NOT count:

  • Running the authors’ official code

  • Copying a blog post implementation

  • Using a library that wraps the method

Why this matters: The delta between “I read this paper” and “I implemented the paper” is where understanding lives. Engineers who regularly do this 3-hour test develop a calibration that paper-readers never acquire. After 10 reproductions, you build an intuition for which architectural choices matter and which are noise.

Paper Management: The Annotation System

Storing papers without annotations is storing noise. The system that works:

Capture Layer

  • Zotero (free, open-source): browser extension captures papers from arXiv with one click. Metadata auto-filled. Syncs across devices.

  • Alternative: direct download to a folder organized by year/topic + Obsidian for notes

Annotation Template (Obsidian / Notion)

## [Paper Title]
**arXiv ID**: [xxxx.xxxxx]  
**Date**: [YYYY-MM]  
**Authors**: [First, Second, et al.]  
**Area**: [Architecture / Training / Alignment / Efficiency / Multimodal]

### The 5 Questions
1. Problem: 
2. Prior work delta: 
3. Core mechanism (2 sentences): 
4. Evidence quality (1-5): 
5. Failure modes: 

### Key Figure
[Description of the figure that captures the core idea]

### Reproduction status
[ ] Pass 1 only  [ ] Pass 2  [ ] Pass 3 + Reproduction

### My opinion
[What I actually think about this paper's significance]

### Connection to my work
[How this relates to what I'm building/studying]

Review Schedule

  • Tag papers by area and re-read tags every 6 weeks. Memory without review is loss.

  • Monthly: pick 1 paper from the past 6 months and do a fresh reproduction. Note what you missed the first time.

What Most People Get Wrong

Reading the methods section first. The methods section is the most technically dense, most carefully written-for-reviewers, and least calibrating section of the paper. If you start there, you absorb the authors’ framing before seeing the evidence. Start with figures and experiments — those are harder to spin.

A second error: treating paper reading as a passive activity. The engineers in the top 5% of field currency are not reading more — they are reading with a pen (or keyboard) in hand, arguing back at the paper, and publishing their disagreements. Passive absorption decays within two weeks. Active critique compounds.

Return to README.md · Next: 02_frontier_papers_2024_2025.md