The Blog Canon¶

Every blog worth your minutes. All URLs verified.

How to use this file¶

Blogs teach what papers can’t: intuition, war stories, aesthetic sense, and the vocabulary of the practitioner community. Papers tell you what; blogs tell you why it matters and how it feels to build it.

The list is organized by phase. Read each one in the phase it maps to — reading Horace He’s brrr post before you understand rooflines wastes it; reading it in Phase 0 while you’re doing napkin math on GPU bandwidth is transformative.

Reading protocol: open the post → read once fast → save to your notes if useful → re-read once carefully with a notebook open → extract one number and one idea into your lab notebook.

Phase 0 — The ethos posts (read these FIRST)¶

Horace He — “Making Deep Learning Go Brrrr From First Principles”¶

URL: https://horace.io/brrr_intro.html
Why: The three-regime framework (compute-bound / memory-bound / overhead-bound) is the mental model for everything in this roadmap. If you internalize this in Phase 0, every subsequent optimization has a home in your head.
Key takeaways: A100 = 312 TFLOPS tensor-core vs 19.5 TFLOPS non-matmul vs 1.5 TB/s HBM. Fusion is the most important compiler optimization. Overhead-bound is a real category and Python is guilty. torch.compile and CUDA graphs exist to attack overhead.
Sequel to read after your Triton work: the follow-up posts on Horace’s blog on torch.compile internals.

kipply — “Transformers Inference Arithmetic”¶

URL: https://kipply.substack.com/p/transformer-inference-arithmetic (also mirrored at https://kipp.ly/blog/transformer-inference-arithmetic/)
Why: The napkin-math bible. Derives from first principles: memory-bound decode latency = 2·P / (N·bandwidth), KV cache = 4·layers·heads·d_head bytes/token, communication bandwidth 300 GB/s.
Read companion: https://kipply.substack.com/p/transformer-param-count — counts every parameter of a transformer, exact formulas.
What to extract: the actual formulas. Type them into a Jupyter notebook. Compute them for Llama-3-8B and Llama-3-70B. Verify against reality.

Chip Huyen — various¶

URL: https://huyenchip.com/blog/
Book: AI Engineering (O’Reilly, 2024) — the product-layer bible.
Why: She sees the industry from the ML platform / eval / production side. Read her posts on evaluation, RAG realities, agentic architectures.
Specific post to read Phase 0: “Multimodality and LLMs” for the survey, and her AI engineering posts.

Phase 2 — GPU architecture + CUDA + Triton¶

Simon Boehm — “How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance”¶

URL: https://siboehm.com/articles/22/CUDA-MMM
Companion repo: https://github.com/siboehm/SGEMM_CUDA
Why: The single best pedagogical treatment of GPU kernel optimization ever written. Ten kernels from naive to 93% of cuBLAS with Nsight measurements at each step. This IS your Phase 2 syllabus.
Also on his blog: https://siboehm.com/articles/22/CUDA-cuBLAS-benchmark and https://siboehm.com/articles/22/data-parallel-training — both worth it.

Hazy Research — “GPUs Go Brrr” (ThunderKittens announce)¶

URL: https://hazyresearch.stanford.edu/blog/2024-05-12-tk
Why: Best modern intro to H100 architecture. Explains WGMMA, TMA, warp specialization, swizzled shared memory in accessible language.
Follow-up posts on the same blog: https://hazyresearch.stanford.edu/blog/2024-10-14-tk-mlsys (ThunderKittens MLSys paper), plus posts on Based, Hedgehog, Just-Read-Twice, and B200/Blackwell content.

Aleksa Gordić — “Inside vLLM: Anatomy of a High-Throughput LLM Inference System”¶

URL: https://www.aleksagordic.com/blog/vllm
Blog index: https://www.aleksagordic.com/blog
Why: THE readable deep-dive into vLLM’s V1 architecture. Covers engine core, PagedAttention, chunked prefill, prefix caching, spec decoding, disaggregation, TP/PP/DP scaling, serving layer, benchmarking.
Length: long. Budget 3 hours. Read once in Phase 4, again after you’ve built your mini engine (it will read differently).
Bonus: Aleksa’s earlier Transformer series (older but foundational).

NVIDIA Developer Blog — selected posts¶

URL: https://developer.nvidia.com/blog/
Must-reads:
- The CUTLASS 3.x posts on CuTe layouts and Hopper GEMM.
- The TMA / cluster / DSMEM Hopper feature announcements.
- The FP8/FP4 training posts.
Filter aggressively. NVIDIA posts a lot; most is product-marketing. The deep-tech ones are worth it.

CUTLASS docs (technically docs, treat as blog)¶

URL: https://github.com/NVIDIA/cutlass/tree/main/media/docs/cpp/cute
CuTe quickstart: https://github.com/NVIDIA/cutlass/blob/main/media/docs/cpp/cute/00_quickstart.md
Why: The layout-algebra docs are unusually good. Save for Phase 2/3 when you go Hopper.

Phase 3 — Attention & fusion¶

Tri Dao — personal + Together AI blog¶

Blog: https://tridao.me/ (personal), https://www.together.ai/blog (Together posts)
Why: The FlashAttention author’s own writings. His FA3 announcement post is essential. Also, several arXiv companion posts on his blog explain FA/Mamba internals.

PyTorch Blog — attention posts¶

URL: https://pytorch.org/blog/
Must-reads:
- “Flash-Decoding for long-context inference” (2023) — the split-KV trick.
- The FlexAttention posts.
- The torch.compile deep dives (Horace He often co-authors).

FlashInfer blog¶

URL: https://flashinfer.ai/
Key post: https://flashinfer.ai/2024/02/02/cascade-inference.html — Cascade Inference (shared-prefix optimization).
Why: Short, dense, per-feature explainers by the FlashInfer team.

Phase 4 — Engines¶

vLLM blog¶

URL: https://blog.vllm.ai
Must-reads: the V1 architecture announcement, the FP8 support posts, the DeepSeek-R1 serving posts, the disaggregated PD posts.
Cadence: new post every 2–4 weeks. Skim as they land.

LMSYS blog (SGLang home)¶

URL: https://lmsys.org/blog/
Must-reads: the original SGLang announcement, the DeepSeek-V3 large-scale serving posts, the RadixAttention deep-dives.

Character.AI engineering blog¶

URL: https://research.character.ai/
Why: Their “Optimizing AI Inference at Character.AI” posts (2024) revealed how they got to 20+ trillion tokens/day: MQA everywhere, cross-layer KV sharing (CLA), hybrid attention (interleaved local+global), int8 quant on weights/activations/KV. This is the highest-throughput production-inference case study publicly available.

Anthropic engineering + research posts¶

URL: https://www.anthropic.com/engineering , https://www.anthropic.com/research
Why: Fewer but high-quality. “Building effective agents” (agentic patterns), Claude context-management posts, MCP protocol — crossover with your Zoho agent work. Interp research (Circuits) is orthogonal but worth awareness.

Anyscale + Ray Serve blog¶

URL: https://www.anyscale.com/blog
Why: Ray Serve is a common orchestration layer above engines. Their LLM-serving posts cover multi-node deployment reality.

Neural Magic (Red Hat AI) blog¶

URL: https://neuralmagic.com/blog/ (redirects to Red Hat now)
Why: They’re the primary vLLM commercial backers post-acquisition. Quant + serving posts are relevant.

RunPod blog¶

URL: https://blog.runpod.io/
Why: Cheaper-GPU-rental perspective. Occasional benchmarks.

Phase 5 — Quantization¶

Tim Dettmers — personal blog¶

URL: https://timdettmers.com/
Must-reads: “Which GPU(s) to Get for Deep Learning” (updated periodically — still the reference), the LLM.int8() and QLoRA companion posts.

HuggingFace blog — quantization series¶

URL: https://huggingface.co/blog
Must-reads: the bitsandbytes intro, the GPTQ integration post, the AWQ post, the QLoRA writeup, the FP8 posts. Search by tag.

The AWQ / SmoothQuant / GPTQ authors’ repos¶

Not exactly blogs, but the READMEs are essential reading. Cite them as references in your bake-off post.

Phase 6 — Distributed training + serving¶

HuggingFace — Ultra-Scale Playbook¶

URL: https://huggingface.co/spaces/nanotron/ultrascale-playbook
Why: The best free end-to-end treatment of 5D parallelism (DP/TP/PP/CP/EP + ZeRO variants) written from thousands of real training runs. This is your Phase 6 bible.
Length: book-length. Read as a book, not a blog post.

Google DeepMind — “How to Scale Your Model”¶

URL: https://jax-ml.github.io/scaling-book/
Why: JAX/TPU framing but the roofline-of-systems thinking is universal and superb. Companion to Ultra-Scale Playbook.

DeepSeek engineering (via their reports + blogs)¶

URL: https://api-docs.deepseek.com/news + arXiv reports
Why: Their V3/V2/R1/Coder reports are the most instructive public documents on co-designing arch/hardware/inference. Their Open Infra Week 2025 posts (DeepEP, DeepGEMM, FlashMLA, DualPipe, EPLB) are gold.

PyTorch Blog — distributed posts¶

FSDP2, TorchTitan, TP/PP integration posts. Same URL: https://pytorch.org/blog/

Kimi (Moonshot AI) tech blog¶

Mooncake architecture, KV-centric serving. Referenced from the Mooncake paper.

Phase 7 — Production / hardware economics¶

SemiAnalysis (Dylan Patel)¶

URL: https://www.semianalysis.com/
Why: The hardware-economics context. Some content paywalled but the free posts are essential for understanding data center economics, GPU supply chain, hyperscaler capex, and why H100/H200/B200 availability matters.

Databricks / Mosaic blog¶

URL: https://www.databricks.com/blog
Why: Training and inference at enterprise scale, including their MPT/DBRX training experience.

Fireworks AI blog¶

URL: https://fireworks.ai/blog
Why: Serving-vendor perspective. Some good disaggregation and spec decoding posts.

Baseten blog¶

URL: https://www.baseten.co/blog/
Why: Deployment perspective, TensorRT-LLM real-world posts.

Perplexity engineering blog¶

URL: https://www.perplexity.ai/hub
Occasional but high-quality serving stack posts.

Ongoing / meta¶

Simon Willison — daily notes¶

URL: https://simonwillison.net/
Why: Curates the entire LLM ecosystem daily with sharp commentary and links. His “weeknotes” and model-release annotations are the fastest way to keep up.
RSS: yes, subscribe.

Sebastian Raschka — Ahead of AI¶

URL: https://magazine.sebastianraschka.com/ (also https://sebastianraschka.com/blog/)
Why: Monthly essays synthesizing new research clearly. Book: Build a Large Language Model (From Scratch) is an alternative to Karpathy for people who prefer books.

Lilian Weng — Lil’Log¶

URL: https://lilianweng.github.io/
Why: Excellent survey-style posts (attention, RLHF, agent design, adversarial). Slower cadence, high quality.

Jay Alammar — The Illustrated ___¶

URL: https://jalammar.github.io/
Why: Illustrated Transformer / Illustrated GPT / Illustrated Retrieval-Augmented Generation. Great for cementing intuition; not a source of frontier info.

Sasha Rush — blog + code¶

URL: https://rush-nlp.com/ , https://github.com/srush
Why: Minimal, focused writeups (GPU puzzles, LLM Training puzzles). His interactive tutorials are excellent.

Yao Fu — blog¶

URL: https://yaofu.notion.site/
Why: Sharp synthesis of scaling / long-context / RL research trends. Chinese-lab-aware perspective.

Nathan Lambert — Interconnects¶

URL: https://www.interconnects.ai/
Why: RLHF/post-training coverage + industry commentary. Weekly.

Phase-mapped summary¶

Phase	Read (in this order)
0	Horace He brrr → kipply arithmetic → Chip Huyen
2	Simon Boehm SGEMM → GPU MODE lecture notes → NVIDIA CUTLASS docs → Hazy TK post
3	Tri Dao personal blog → PyTorch Flash-Decoding → FlashInfer Cascade → Hazy TK-2 posts
4	Aleksa Gordić Inside vLLM → vLLM blog → LMSYS SGLang blog → Character.AI inference posts
5	Tim Dettmers GPU/quant → HF quant series → AWQ/GPTQ/SmoothQuant READMEs
6	HF Ultra-Scale Playbook → How to Scale Your Model → DeepSeek Open Infra Week posts
7	SemiAnalysis → Fireworks/Baseten → Character.AI (revisit through SLO lens)
Ongoing	Simon Willison daily, Sebastian Raschka monthly, Lambert weekly

Anti-canon (blogs to skip)¶

Medium articles from unknown authors summarizing papers. 90% wrong or vibes-only.
Substack thought-leader posts about “the future of AI” — read the primary sources instead.
Any blog whose top posts are “prompt engineering tricks.” You are past that layer.
Vendor blogs whose content is 80% product pitch. (Filter aggressively; the good ones above are exceptions.)