Frontier Papers 2024–2025: The Practitioner’s Reading List¶
There are roughly 100–200 new ML papers published on arXiv every single day. That is not a reading problem — it is a filtering problem. This document solves the filter for the period that matters most to a practitioner right now: 2024–2025. These are not the most academically cited papers; they are the papers whose ideas are already reshaping production systems, interview questions, and job descriptions at AI labs.
For each paper, the format is: arXiv ID, date, the core insight in 2 sentences, and — most importantly — why a working ML engineer should care. “Why it matters” is not a summary. It is the answer to: what changes in your practice, your architecture decisions, or your evaluation criteria if you internalize this paper?
Architecture Innovations¶
1. Mixtral of Experts (Mistral AI)¶
arXiv: 2401.04088 | Date: January 8, 2024
Core insight: Replace each dense FFN layer with 8 expert networks; route each token to only 2 of them using a learned gating function. Total parameters: 47B. Active parameters per forward pass: ~13B (8×7B parameter-class model with 2-of-8 expert selection).
Why it matters: This was the first open-weight model to demonstrate that Mixture-of-Experts scales to competitive performance against much larger dense models. It outperformed Llama 2 70B and GPT-3.5 on most benchmarks while requiring only 2/8 of the FFN compute per token. If you are doing anything with inference cost optimization or model serving, the MoE architecture is now a baseline consideration — not an exotic research idea. The paper also made sparse expert routing accessible: the gating mechanism is simpler than most practitioners expect.
Key number: 47B total params, ~13B active per token. Inference cost ≈ 12B dense model. Quality ≈ 70B dense model.
2. DeepSeek-V2: Multi-head Latent Attention + Fine-grained MoE¶
arXiv: 2405.04434 | Date: May 2024
Core insight: Multi-head Latent Attention (MLA) compresses key-value pairs into a low-dimensional latent space, then reconstructs them — achieving 93.3% reduction in KV cache size vs. standard MHA with negligible performance loss. Combined with fine-grained MoE (more experts, smaller each), DeepSeek-V2 trains 42.5% cheaper than its predecessor.
Why it matters: KV cache is the dominant inference memory bottleneck for LLM serving at scale. If you are deploying LLMs in production, KV cache memory determines how many concurrent requests you can serve and at what context length. MLA is a direct solution to this. This paper is required reading for anyone working on LLM inference infrastructure. The 93.3% KV cache reduction is not a rounding error — it is a fundamental efficiency improvement that survived into DeepSeek-V3 and R1.
Key number: 93.3% KV cache reduction vs MHA. 42.5% cheaper training per token than DeepSeek-67B.
3. DeepSeek-V3¶
arXiv: 2412.19437 | Date: December 26, 2024
Core insight: 671B total parameters, 37B active per token (MoE). Trained on 14.8 trillion tokens. Introduces auxiliary-loss-free load balancing (a cleaner signal than the entropy-based load balancing used previously) and multi-token prediction (predict the next N tokens simultaneously, not just next 1) as an auxiliary training objective.
Why it matters: DeepSeek-V3 rivals GPT-4-class models and was reportedly trained for approximately $6M in compute — orders of magnitude cheaper than comparable closed-source efforts. This paper redefined what “expensive to train” means and put Chinese open-source models directly on par with frontier closed-source systems. The multi-token prediction auxiliary objective is practically useful for inference speedup. Auxiliary-loss-free load balancing is a cleaner engineering solution than the entropy regularization used in prior MoE work. Read this paper to understand the current state of MoE at scale.
Key number: 671B total / 37B active / 14.8T tokens / ~$6M training cost.
4. Mamba: Linear-Time Sequence Modeling with Selective State Spaces¶
arXiv: 2312.00752 | Date: December 2023 (exploded in 2024)
Core insight: Selective state space models (SSMs) whose parameters are functions of the input — allowing the model to selectively remember or forget information based on content, unlike fixed SSMs. Hardware-aware parallel scan replaces sequential state updates, enabling training at transformer-competitive speed.
Why it matters: Mamba provided the first credible architecture that scales to transformer-competitive performance on language tasks with linear (not quadratic) complexity in sequence length. For long-sequence applications (genomics, audio, very long documents), Mamba is a genuine alternative. The 2024 landscape saw Mamba2 and hybrid Mamba-Transformer models emerge. The key limitation to know: Mamba has documented weaknesses in input copying tasks and multi-step retrieval — weaknesses that transformers handle trivially. Understanding why helps you understand what attention is actually doing that SSMs cannot easily replicate.
Key number: Linear complexity O(N) vs quadratic O(N²) for attention. Known failure: in-context retrieval tasks.
Training Innovations¶
5. Flash Attention 2¶
arXiv: 2307.08691 | Date: July 2023 (dominant in 2024 practice)
Core insight: Attention computation tiled to operate within SRAM rather than repeatedly reading from HBM (GPU high-bandwidth memory). Forward pass and backward pass recomputed during backprop rather than stored. Result: memory bandwidth reduced from O(N²) to O(N); wall-clock speed 2–4× faster than standard attention for long sequences.
Why it matters: Flash Attention 2 is not optional reading for production ML — it is already embedded in every major training framework. But understanding why it works (IO awareness: the bottleneck is memory bandwidth, not FLOPs) changes how you think about ML system design. The principle — profile the actual bottleneck rather than optimizing FLOPs — applies beyond attention. Tri Dao’s follow-on work extends this to other components. If you are doing anything with long-context training (>4K tokens), Flash Attention 2 is the reason it is tractable.
Key number: 2–4× faster wall-clock speed. Memory: O(N) vs O(N²). Now standard in PyTorch 2.x via scaled_dot_product_attention.
6. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints¶
arXiv: 2305.13245 | Date: May 2023 (adopted across all 2024 LLMs)
Core insight: Grouped Query Attention (GQA) is a middle ground between Multi-Head Attention (MHA, 1 KV head per Q head) and Multi-Query Attention (MQA, 1 KV head shared across all Q heads). GQA groups queries and shares KV heads within each group — achieving near-MQA inference speed with near-MHA quality.
Why it matters: GQA is in Llama 3, Mistral, Gemma, and essentially every serious LLM released in 2024. Understanding GQA is understanding the KV cache trade-off that every modern LLM architect has made. If you are fine-tuning, evaluating, or serving any of these models, you need to understand why GQA exists and what it costs. The paper also introduces a technique to convert existing MHA checkpoints to GQA without full retraining — practically useful if you are working with older base models.
Alignment Innovations¶
7. Direct Preference Optimization (DPO)¶
arXiv: 2305.18290 | Date: May 2023 (dominant training method in 2024)
Core insight: The RLHF objective — maximize reward while staying close to the reference policy — can be reformulated directly as a classification loss on preference pairs, eliminating the need for a separate reward model and PPO training loop.
Why it matters: DPO reduced the cost and complexity of preference alignment by roughly 10×. The 2024 fine-tuning landscape is built on DPO variants: IPO, KTO, ORPO. If you are doing RLHF or instruction tuning, DPO is the baseline you are measured against. The key insight is that the reward function is implicitly defined by the policy — you do not need to learn it separately. Understanding this derivation is a common senior ML interview question at AI-focused companies.
8. DeepSeekMath / GRPO: Group Relative Policy Optimization¶
arXiv: 2402.03300 | Date: February 2024
Core insight: GRPO eliminates the critic/value network from PPO by estimating the baseline from the group of outputs sampled for the same prompt. Each output’s advantage is computed relative to the group mean reward — no value network, no advantage estimation network.
Why it matters: GRPO is the training algorithm behind DeepSeek-R1 — the reasoning model that matched o1-class performance for reportedly under $300K in RL compute. Understanding GRPO is understanding why the “reasoning via RL” paradigm became tractable for teams without Google/OpenAI compute. For practitioners interested in training reasoning models, GRPO is the direct successor to PPO for LLMs. The removal of the critic network is not just a cost reduction — it avoids the credit assignment instability that makes PPO difficult to tune for long-horizon tasks.
Multimodal¶
9. LLaVA / LLaVA-1.5¶
arXiv: 2304.08485 (LLaVA) / 2310.03744 (1.5) | Date: April 2023 / Oct 2023 (dominant reference in 2024)
Core insight: Visual instruction tuning: connect a frozen CLIP vision encoder to a frozen/fine-tuned LLM via a lightweight projection layer; generate visual instruction-following data using GPT-4 to describe images; fine-tune end-to-end. LLaVA-1.5 shows that a simple MLP projection outperforms complex cross-attention bridges.
Why it matters: LLaVA established the recipe that most open multimodal models follow in 2024. The insight that you do not need architectural complexity to bridge vision and language — just a projector + instruction data quality — is directly actionable if you are building multimodal applications. LLaVA-1.5’s result that an MLP projector beats cross-attention was unexpected and important: it suggests alignment is primarily a data problem, not an architecture problem.
10. Gemma 2 (Google DeepMind)¶
arXiv: 2408.00118 | Date: August 2024
Core insight: Alternating local (sliding window) and global attention layers, logit soft-capping (preventing logit explosion without clipping), and knowledge distillation from a larger teacher model — achieving competitive performance at 9B and 27B scales with much lower inference cost.
Why it matters: Gemma 2 is a practitioner-friendly model with Apache 2.0-equivalent licensing and strong open benchmarks. The logit soft-capping technique is a stabilization trick worth knowing. More importantly: the distillation recipe is detailed and reproducible — Gemma 2 27B was distilled from a larger model, and the paper explains how. If you are working on model compression or distillation pipelines, this is a reference implementation from a major lab.
Efficiency: Quantization, Distillation¶
11. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers¶
arXiv: 2210.17323 | Date: Oct 2022 (but the 2024 ecosystem is built on it)
Core insight: One-shot weight quantization using approximate second-order information (Hessian-based). Quantizes LLMs to 4-bit and 3-bit with minimal perplexity loss, in a single forward pass over calibration data.
Why it matters: GPTQ + GGUF (llama.cpp) + AWQ are the three quantization methods that dominate production LLM deployment in 2024. If you are running LLMs on consumer hardware or deploying at the edge, you are almost certainly using one of these. Understanding how GPTQ works — specifically, that it minimizes quantization error layer-by-layer using Hessian information — is the difference between blindly applying quantization and being able to diagnose quality degradation.
Reasoning¶
12. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Wei et al.)¶
arXiv: 2201.11903 | Date: January 2022 (foundational; all 2024 reasoning work builds on this)
Core insight: Simply including intermediate reasoning steps (chain of thought) in few-shot examples causes LLMs to emit intermediate steps themselves, dramatically improving performance on arithmetic, symbolic reasoning, and commonsense tasks.
Why it matters: Every reasoning model — o1, o3, DeepSeek-R1, Qwen-QwQ — is a direct extension of this observation. The insight is that LLMs can perform multi-step reasoning if they are given space (tokens) to do so. The 2024–2025 o1-style paradigm is: train the model to generate chain of thought at inference time as part of its output, rather than receive it as a prompt. If you want to understand what o1 is doing, start here.
13. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning¶
arXiv: 2501.12948 | Date: January 2025
Core insight: Train a reasoning model using GRPO with a verifiable reward signal (rule-based: correct/incorrect for math/code). Start from DeepSeek-V3. The model develops chain-of-thought reasoning behavior spontaneously — without SFT on human-written reasoning traces. Achieves o1-comparable performance on AIME 2024 and Codeforces.
Why it matters: DeepSeek-R1’s public release caused measurable stock price moves at US AI companies. It demonstrated that o1-class reasoning is achievable with open weights and public methodology, at a fraction of the cost. For practitioners: the verifiable reward signal insight is the key — math and code allow automatic correctness checking, which enables scalable RL training without human feedback. The paper details the cold-start instability problem (R1-Zero) and how supervised warm-up solves it. This is the most important alignment paper of early 2025.
Key number: ~$300K reported RL compute cost. AIME 2024: 79.8% (comparable to o1-preview).
How to Prioritize These 13 Papers¶
If you have limited time, read them in this order based on cumulative leverage:
Flash Attention 2 (understanding the infrastructure all else runs on)
GQA (it’s in every model you touch)
DPO (it’s in every fine-tuning pipeline)
Mixtral (MoE fundamentals)
DeepSeek-V2 (KV cache economics)
DeepSeek-R1 (current frontier)
Chain-of-Thought (reasoning foundations)
GRPO (the algorithm behind R1)
Mamba (understanding the SSM alternative)
DeepSeek-V3 (scale + cost efficiency)
LLaVA-1.5 (multimodal baseline)
GPTQ (deployment fundamentals)
Gemma 2 (distillation reference)
What Most People Get Wrong¶
Reading the “why it matters” summary and thinking they’ve read the paper. Summaries — including this one — are maps, not territory. A summary of Flash Attention tells you it’s faster. Reading the paper tells you why memory bandwidth is the bottleneck and what tiling strategy solves it. That second level of understanding is what survives in an interview when someone asks “explain Flash Attention from first principles” and you can actually derive the IO cost argument.
Return to 01_how_to_read_papers.md · Next: 03_open_source_contribution.md