05 — Phase 0 Projects

Three concrete deliverables. Each is small enough to finish in a week of evenings, big enough to be portfolio-worthy. Complete all three before moving to Phase 1’s heavier lifts (though you should be watching Karpathy videos in parallel).

Project 0.1 — Tiled Matmul Speedup (C++)

Time budget: 6–10 hours over one week.

Goal: Prove to yourself, with numbers, that the memory hierarchy dominates naïve arithmetic. Establish the “predicted-vs-measured” habit.

Steps:

  1. Write matmul_naive.cpp: the triple-loop fp32 matmul. Compile with -O3 -march=native -fno-tree-vectorize (disable auto-vectorization to keep the story clean).

  2. Write matmul_tiled.cpp: same signature, tiled with BM=BN=BK=32 (fits in L1). Same compile flags.

  3. Write bench.cpp that runs both on 512×512, 1024×1024, 2048×2048 fp32 matrices. Report GFLOPS.

  4. Under perf stat -e cycles,instructions,cache-references,cache-misses,LLC-load-misses, run both. Save the output.

  5. Now enable -O3 -march=native with auto-vectorization. Rerun. Note the improvement from SIMD.

  6. Compare to a call to cblas_sgemm (via OpenBLAS). Report the ratio.

Acceptance criteria:

  • Tiled ≥ 8× naïve on 1024² fp32 (no vectorization).

  • With -march=native vectorization, tiled ≥ 4× the non-vectorized tiled version.

  • Your bench.cpp uses proper warmup and reports median (not mean) of ≥5 runs.

  • perf output shows LLC-load-misses drop by ≥5× from naive to tiled.

  • You’ve predicted each speedup on paper before measuring, and your prediction is within 30% of measured.

  • README explains WHY each optimization worked in terms of cache-line traffic.

  • Committed to a public repo with a one-page write-up.

Bonus (do this): port the tiled version to fp16 with _Float16 and measure. Then port to Eigen for comparison. Then, once you’re in Phase 2, rewrite this on the GPU and observe how the same optimizations map.

Project 0.2 — The Numerics Notebook

Time budget: 4–6 hours.

Goal: Own the bit-level truth of every floating-point format you’ll encounter for the next three phases.

Steps:

  1. In a Jupyter notebook, use struct.pack/struct.unpack (Python) or bit_cast (C++) to decode any fp32 into [sign, exponent, mantissa]. Verify with math.frexp.

  2. Implement bf16 as truncated fp32 (drop lower 16 bits of the mantissa — that’s it). Verify against torch.bfloat16.

  3. Implement fp8-e4m3 encode/decode in pure Python. Compare a range of values to what torch.float8_e4m3fn produces (PyTorch 2.1+).

  4. Implement a symmetric per-channel int8 quantizer for a fp32 weight tensor: compute scale = max(abs(w)) / 127, quantize, dequantize, report the MSE.

  5. Implement a group-wise (group=128) int4 quantizer with one fp16 scale per group. Report MSE. Compare to per-tensor int4 (bad) and per-channel int4.

  6. Chart the smallest positive normal value, largest finite value, and machine epsilon for each format. Print a table.

  7. Then: re-derive online softmax. Implement softmax_online in numpy that consumes one element at a time and matches scipy.special.softmax after the pass. Test with an adversarial input that overflows naive softmax.

Acceptance criteria:

  • fp8-e4m3 encoder round-trips exactly to torch.float8_e4m3fn for at least 100 random fp32 inputs in-range.

  • int8 per-channel quantization MSE < int8 per-tensor MSE on a real weight matrix (grab any Llama tensor).

  • Online softmax matches scipy.special.softmax to within 1e-6 on a test with x = [100, 100.5, -50, 200] (where naive softmax overflows).

  • Table of format properties printed and committed as formats.md.

  • You can, from memory, draw all six format bit-layouts (test yourself weekly).

Project 0.3 — Latency & Bandwidth Atlas of Your Machine

Time budget: 3–5 hours.

Goal: Fill in Jeff Dean’s latency table for your actual hardware, so numbers stop being abstract.

Steps:

  1. CPU cache latencies: use lat_mem_rd from lmbench (brew install lmbench on macOS, apt install lmbench on Linux) to sweep working-set size from 1 KB to 1 GB. Plot latency vs size. You should see three plateaus (L1, L2, LLC) and a wall (DRAM).

  2. CPU memory bandwidth: use STREAM (John McCalpin’s classic: https://www.cs.virginia.edu/stream/) to measure copy/scale/add/triad bandwidth. Compare to your CPU’s spec.

  3. PCIe host↔device bandwidth: use nvbandwidth (https://github.com/NVIDIA/nvbandwidth) or a simple cudaMemcpy benchmark. Sweep transfer size from 1 KB to 1 GB.

  4. GPU HBM bandwidth: run bandwidthTest from CUDA samples, or write a simple copy kernel. Compare to spec (e.g., 3090 = 936 GB/s theoretical, actual ~800–850 GB/s).

  5. GPU shared memory bandwidth (optional): run one of the GPU MODE micro-benchmarks (when you get to Phase 2) or a stream benchmark. On an H100 shared memory sustains ~30 TB/s per SM.

Acceptance criteria:

  • latency_table.md filled with your numbers: L1 (~cycles), L2, LLC, DRAM, PCIe small, PCIe large, HBM read, HBM write.

  • STREAM copy bandwidth within 20% of the theoretical peak of your DDR.

  • cudaMemcpy bandwidth reported for pinned vs non-pinned. Difference explained.

  • HBM_write_bandwidth / HBM_read_bandwidth ratio noted (usually near 1).

  • The plot of latency vs working-set size shows plateaus at your cache sizes.

Phase 0 Portfolio Manifest

At the end of Phase 0, you should have a single public repo (name suggestion: inference-eng-phase0 or your own) with:

01-matmul/
  matmul_naive.cpp
  matmul_tiled.cpp
  bench.cpp
  perf_results.txt
  README.md
02-numerics/
  formats.ipynb
  formats.md
  softmax_online.py
  softmax_tests.py
03-latency-atlas/
  latency_table.md
  stream_output.txt
  bandwidth_plot.png
  script.sh
04-cache-lab/            # from Section 01
  parts_A_and_B/
05-notes/
  brrr_notes.md          # Your notes on Horace He's essay
  csapp_ch5_ch6_notes.md
  bit_layouts.md         # The drill drawings, dated to prove weekly practice

A README at the root ties them together and links to your public write-ups (Twitter/X, blog, or GitHub). A single tweet-thread walking through the matmul speedup and roofline analysis usually generates useful feedback from GPU MODE folks.

The Predicted-vs-Measured Lab Notebook

Start your habitat now, before Phase 1 raises the stakes. A simple Markdown file, one entry per experiment:

## 2026-07-05: naive matmul on 1024x1024 fp32

**Hypothesis:** Naive matmul is dominated by L2/LLC misses on B (column-major access).
**Predicted GFLOPS:** ~1.5 (assuming ~200 ns DRAM latency dominates every K iteration).
**Measured GFLOPS:** 1.8.
**Gap:** Prediction slightly pessimistic; hardware prefetcher is catching some of the accesses.
**Nsight/perf evidence:** `LLC-load-misses` = 4.2e7; `cache-references` = 2.1e9; miss rate 2%.
**Next action:** Tile with BM=BN=BK=32; predict 15–20× speedup from working-set fitting L1.

Every experiment. Every phase. This notebook is what you’ll show at your first onsite study.