Phase 0 — Systems & Math Foundations¶

Duration: Weeks 1–10 (partly parallel with Phase 1) Budget on the 13-month viking timeline: ~8–10 weeks of focused work at 15–20 hrs/week. Prerequisite: You already ship Python day-to-day at Zoho and think in services. Good. This phase closes the substrate.

Why Phase 0 Exists¶

You cannot write a CUDA kernel if you cannot draw the CPU cache hierarchy from memory. You cannot debug a numerically unstable softmax if you cannot recite the bit layout of bf16. Everything downstream — Triton, FlashAttention, PagedAttention, FP8 quantization — is built on assumptions this phase makes explicit. If you skip Phase 0, kernel work in Phases 2–3 will feel like magic, and magic is what tutorial-hoppers mistake for expertise.

The organizing insight: inference engineering is arithmetic about silicon. Every optimization is a story about bytes moved, FLOPs done, cycles wasted. Phase 0 gives you the units — cache lines, cycles, exponent bits, roofline coordinates — that every later phase will manipulate.

The Five Subtopics¶

#	File	What you’ll own
1	`01_computer_systems.md`	CS:APP cache/malloc lab; `perf`; flamegraphs; latency numbers table with GPU numbers appended
2	`02_cpp_for_cuda.md`	RAII, `const`, templates at reading level, CMake at survival level, one llama.cpp file annotated line-by-line
3	`03_python_performance.md`	GIL truths, `asyncio` at engine-loop depth, NumPy strides/views, `torch.profiler`, `py-spy` flame graphs
4	`04_numerics_math.md`	Tiled matmul (paper → C++ → 10× speedup measured), online softmax derived from scratch, bit layouts of fp32/bf16/fp8-e4m3/e5m2/int8/int4 memorized, sampling math
5	`05_projects.md`	Three concrete deliverables with acceptance criteria

Exit Criteria (all three, no negotiation)¶

You are done with Phase 0 when you can:

Prove the cache. Write naive and tiled CPU matmul in C++. Measure ≥8× speedup with the tiled version on a matrix that exceeds L2. Explain each cache-line miss on paper. Attach perf stat output showing LLC-load-misses collapse.
Draw the numbers. From memory, on a whiteboard: bit layouts of fp32, bf16, fp8-e4m3, fp8-e5m2, int8. Explain why bf16 won for training (dynamic range = fp32’s exponent) and why fp8-e4m3 is winning for inference forward passes (finer mantissa where activations live). Explain in one sentence why e5m2 exists (gradients need range).
Derive online softmax. Given x = [x_1, ..., x_N] and no prior knowledge, derive the streaming softmax algorithm that maintains a running max m and running sum l such that after one pass you have numerically stable softmax(x). Show the rescaling identity l_new = l_old * exp(m_old - m_new) + exp(x_i - m_new). This is FlashAttention’s inner loop; if you don’t own it now, Phase 3 will hurt.

If you cannot do all three cold, do not move on. The phases compound.

The Meta-Habit¶

For every experiment in this phase and every phase after, write in a lab notebook:

Hypothesis:  <what I expect and why, in numbers>
Predicted:   <napkin arithmetic result>
Measured:    <actual number, with the benchmark script committed>
Gap:         <why they differ, honestly>

The predicted-number habit is the entire skill of this field. Start it now on toy problems (matmul speedups) so it’s automatic by the time you’re predicting decode throughput for a 70B on 4×H100.

Reading Order Within Phase 0¶

Weeks 1–3: 01_computer_systems.md + 04_numerics_math.md in parallel (they reinforce). Weeks 3–5: 02_cpp_for_cuda.md and start 05_projects.md project #1 (tiled matmul). Weeks 5–7: 03_python_performance.md while you start Phase 1 (nanoGPT in parallel). Weeks 7–10: Finish projects. Write up. Exit.

You should be reading Karpathy videos (Phase 1) from week 3 onwards — the phases overlap deliberately.