Rung 7: Paper Reproduction¶

Month 13 | Phase 6: Research | ~40–55 hours total

Reproducing a research paper is the highest-fidelity test of whether you can actually engage with the scientific literature — not just read papers for the ideas, but understand them at implementation depth. The distinction between a tutorial reproduction and an independent implementation is fundamental: a tutorial follows instructions, an independent implementation forces you to resolve the paper’s ambiguities, implement what the paper describes rather than what the released code does, and produce original analysis of where the results align and where they diverge. This rung proves you can bridge theory and practice at a research level.

Paper Selection Criteria¶

Choosing the right paper is as important as implementing it well. A paper that is too simple signals superficiality. A paper that is too complex produces an incomplete reproduction that proves nothing. The following criteria define the target zone:

Required Criteria (all must be true)¶

Criterion	Why It Matters
Published 2023–2025	Demonstrates you’re reading current literature, not just classics
Has an official code release	You can verify your implementation against the reference (but must not copy it)
Reproducible on consumer hardware	Must be trainable on Kaggle/Colab Pro/RunPod within reasonable compute budget (< $50)
Introduces a specific, concrete mechanism	Must have something to implement beyond “we fine-tuned a big model”
Not already over-reproduced	Avoid papers with 50+ existing reproduction repositories

Strongly Recommended Papers (2023–2025 range, appropriate scope)¶

Architecture / Efficiency:

RetNet (Retention Networks, Microsoft 2023) — attention alternative with parallel/recurrent duality
Mamba (Selective State Space Models, Gu & Dao 2023) — SSM with selective state space, tractable on small scale
GQA (Grouped Query Attention, Ainslie et al. 2023) — grouped key-value heads, important inference optimization

Training / Optimization:

DoRA (Weight-Decomposed Low-Rank Adaptation, Liu et al. 2024) — decomposed PEFT method, implementable as alternative to LoRA
GaLore (Gradient Low-Rank Projection, Zhao et al. 2024) — memory-efficient training, interesting theoretical contribution
Spectrum (targeted layer training, 2024) — selective layer freezing during fine-tuning

LLM Evaluation / Alignment:

Self-RAG (Asai et al. 2023) — retrieval with self-reflection tokens, implementable at small scale
Constitutional AI mechanics (Anthropic 2022, but mechanisms are still novel to implement)
RLHF reward modeling: reproduce the reward model training loop on a specific task with a small dataset

Diffusion / Generation:

DDIM (Song et al. 2020) — deterministic sampling, mathematically rich, implementable on MNIST/CIFAR
Latent Consistency Models (Song et al. 2023) — distillation of diffusion, interesting tradeoffs
Flow Matching (Lipman et al. 2022) — alternative to diffusion, cleaner mathematics

Multimodal (if Phase 3 work covered ViT):

CLIP on small scale (reproduce the contrastive training loop on a small image-text dataset)
LLaVA visual instruction tuning (reproduce on a small VQA subset)

How to Choose¶

Select a paper where you can answer “yes” to all three questions:

“Do I understand the core mechanism well enough to explain it in 2 minutes without notes?”
“Can I implement the key novel component from the paper description alone, without reading the released code first?”
“Is there something interesting to say about the results — either confirming, contradicting, or extending the paper’s claims?”

If you can’t answer yes to all three after reading the paper twice, choose a different paper.

The Independence Constraint¶

This is the single most important rule for this rung and the most commonly violated:

You must implement the paper’s core mechanism from the paper description (methods section + appendix), not from the released code.

The allowed workflow:

Read the paper fully. Implement from scratch.
Run your implementation and collect results.
Compare your results to the paper’s reported results.
After you have your own results: read the released code to understand discrepancies.
Document what the code revealed that the paper didn’t say explicitly.

The prohibited workflow:

Read the paper.
Read the released code.
Adapt the released code to your setup.
Call it a reproduction.

Step 5 in the allowed workflow — “what the code revealed that the paper didn’t say” — is often the most interesting finding in a reproduction. Papers omit implementation details that turn out to matter significantly. Finding these omissions is a contribution.

What the Analysis Must Include¶

The reproduction without analysis is an implementation exercise. The analysis is what makes it a portfolio artifact. Your ANALYSIS.md or equivalent must address all of the following:

1. Implementation Decisions (every ambiguity resolved)¶

For every place where the paper was ambiguous or underspecified, document:

What the ambiguity was (quote the relevant paper text)
What implementation decision you made and why
What happened when you tried the alternative (run the experiment)

Example: “The paper states ‘we use layer normalization before the attention layer’ but does not specify whether the final projection also has layer normalization. We tried both and found [X].”

2. Quantitative Comparison Table¶

| Experiment | Paper Reports | Our Reproduction | Delta | Notes |
|------------|--------------|-----------------|-------|-------|
| [Task A]   | XX.X%        | XX.X%           | +/-X% | [Explanation] |
| [Task B]   | XX.X%        | XX.X%           | +/-X% | [Explanation] |

If your numbers don’t match the paper exactly, this is expected and valuable. Explain why they might differ: different random seeds, compute budget, dataset version, implementation details the paper didn’t specify.

3. What the Paper Didn’t Say¶

Document at least 2 implementation details that you discovered were critical but were not explicitly stated in the paper. These are the most valuable findings in any reproduction. Common examples:

Specific initialization scheme that matters for convergence
Learning rate warmup duration that significantly affects results
Data augmentation that the paper mentions in a footnote but doesn’t quantify
Numerical stability trick in the implementation (epsilon values, gradient clipping threshold)

4. Compute and Resource Analysis¶

How much does this approach actually cost to run?

GPU hours required to reproduce key results
Cost estimate on various cloud platforms (Kaggle free, Colab Pro, RunPod A100)
Memory footprint at various scales
Comparison to what the original paper reported (papers often use institutional compute far beyond individual access)

5. One Genuine Extension or Test¶

Run one experiment that the original paper did not run. Options:

Apply the method to a different domain or dataset than the paper used
Compare the method to a recent competitor the original paper couldn’t have compared to
Ablate a component the paper didn’t ablate
Test a hypothesis the paper’s results suggested but didn’t pursue

This extension does not need to be a breakthrough. It needs to be genuine — something you decided to test because of what you observed, not because a tutorial told you to.

Where to Publish¶

Primary: GitHub repository with:

Clean, well-structured implementation code
ANALYSIS.md with all sections above
Training scripts that reproduce your results in < 5 commands
results/ directory with your generated plots and metrics
figures/ with comparison figures (paper’s results vs. yours, if available)

Secondary: Blog post on your technical writing platform summarizing the reproduction findings for a non-specialist audience. This blog post should be accessible to someone who has not read the paper — explain what the paper proposes, what you found when you implemented it, and what the discrepancies mean.

Optional but high-signal: If your reproduction reveals a significant implementation detail the paper omitted or a reproducibility gap, consider submitting to the ML Reproducibility Challenge (held annually, ~October-December) or posting to Papers With Code as a linked reproduction. This connects your work to the broader reproducibility ecosystem and creates external validation.

Time Estimate¶

Task	Estimated Hours
Paper reading (full, with notes)	4–6
Implementation from paper description	12–16
Initial training runs and debugging	6–8
Analysis and comparison to paper results	4–5
Extension experiment	4–5
ANALYSIS.md writing	4–5
Blog post writing	3–4
README + code cleanup	2–3
Total	39–52 hours

What Weakens This Rung¶

Reproducing a paper that everyone has reproduced: if there are already 50+ GitHub repos implementing Attention Is All You Need, your 51st adds nothing. The paper selection must be non-trivial.
Adapting released code instead of implementing from scratch: immediately visible to anyone who reads both your code and the original. The commit timestamps relative to your stated methodology will also reveal this.
Results that exactly match the paper with no analysis: paradoxically, exact matches are suspicious. Either you copied something, or you got very lucky with hyperparameters. The honest path includes acknowledging where your results diverge and explaining why.
Analysis that only confirms the paper’s claims: a reproduction that says “we confirmed X, Y, and Z from the paper” without any finding of ambiguity, discrepancy, or extension is not analysis — it is validation. Validation is useful but shallow.
No compute budget analysis: claiming to reproduce a result that required 128 A100s in a paper using a Colab T4 is implausible. If you’re reproducing at a smaller scale, document what you reproduced, what you couldn’t reproduce, and why.
Blog post that is a paper summary, not a reproduction narrative: summarizing the paper’s claims is not the same as writing about your experience implementing them. The blog post must contain your findings, your surprises, and your questions — not a paraphrase of the abstract.

Return to README.md · Previous: 06_rung_6_production_ml_system.md · Next: 08_rung_8_technical_writing.md