02 — The Lab Notebook

Predict the number. Measure the number. Explain the gap. That’s the entire skill.

Why this file matters more than any other technical resource in the roadmap

Everything you’re going to learn in the next 13 months — CUDA occupancy, PagedAttention, speculative decoding, tensor parallelism, W4A16, chunked prefill — is downstream of one habit:

Before you run the benchmark, write down what number you expect and why.

That’s it. That is the fluency test. The people who can look at “Llama-3-8B, seq=2048, batch=16, RTX 3090” and instantly say “decode should be ~65 tok/s, bound by HBM at ~700 GB/s effective, KV cache dominates memory” — they didn’t memorize that. They developed the reflex by writing down predictions, being wrong, and explaining why for years.

You’re going to do that starting sprint 1.

The /lab_notes/ folder

At the root of your workspace, a git-tracked folder:

/lab_notes/
├── 2026-08-05_llama3_8b_decode_prediction.md
├── 2026-08-06_max_batch_size_24gb_8k_ctx.md
├── 2026-08-08_triton_softmax_bandwidth.md
├── ...
├── surprise_log.md                       # see below
└── README.md                             # index, updated weekly

One markdown file per experiment. Not per sprint. Not per day. Per experiment. An experiment is any moment where you have a specific question and a specific number you’re going to measure.

Commit these. All of them. Even the embarrassing ones. Especially the embarrassing ones.

The template (copy exactly)

# [Experiment title — one line, searchable]

**Date:** YYYY-MM-DD
**Hardware:** RTX 3090, driver 550.xx, CUDA 12.4, PyTorch 2.4
**Related sprint:** sprint_07
**Related file/PR:** kernels/softmax_v2.py

## 1. Hypothesis (one sentence)

> If I fuse the softmax reduction with the mask, memory traffic drops from
> 3x tensor size to 2x, so I expect ~1.4x speedup on the memory-bound regime.

## 2. Arithmetic — predicted number

Show your work. Not "I think it'll be faster." Show the numbers.

- Input tensor: (B=16, H=32, S=2048, S=2048) fp16 = 16 * 32 * 2048 * 2048 * 2 bytes
  = 4.29 GB
- Baseline memory traffic: 3 * 4.29 GB = 12.87 GB per call
- Fused traffic: 2 * 4.29 GB = 8.58 GB
- RTX 3090 effective HBM bandwidth: ~700 GB/s (80% of theoretical 936)
- Predicted baseline time: 12.87 / 700 = 18.4 ms
- Predicted fused time:    8.58  / 700 = 12.3 ms
- **Predicted speedup: 1.50x**
- **Predicted absolute: 12.3 ms**

## 3. Measurement — actual number

- Baseline (10-run median, warmed): 19.1 ms
- Fused (10-run median, warmed):    14.7 ms
- **Measured speedup: 1.30x**
- **Measured absolute: 14.7 ms**

Benchmark script: `bench/softmax_fused.py` (committed at commit abc123).
Full run log: `bench/logs/2026-08-08_softmax.log`.

## 4. Gap explanation

Predicted 12.3 ms, measured 14.7 ms — a 19% miss.

Where did the 2.4 ms go?

- Kernel launch overhead: not accounted for. Nsight Systems shows ~40 μs
  per launch, 2 launches per fused call → negligible (~0.08 ms). Not it.
- Register pressure: ncu says achieved occupancy dropped from 62% to 41%
  when I fused. That's the real cost. Fewer warps in flight → less latency
  hiding on HBM.
- L2 residency: partial. The re-read of the mask that I "eliminated" was
  actually mostly L2-hit in the baseline. So the memory saving was smaller
  than raw-bytes math suggested.

## 5. Lesson (one sentence, memorable)

> Bytes-through-HBM is a ceiling, not the actual traffic — L2 residency
> can hide 20-40% of "eliminated" reads.

## 6. Follow-up questions

- Would a smaller block size (128 vs 256) restore occupancy without losing the fusion benefit?
- How does this change at seq=4096 where L2 pressure is worse?

Every field is mandatory. If you find yourself wanting to skip “predicted number”, stop. That’s the exact reason this template exists.

Worked example 1: Predicting Llama-3-8B decode tok/s on a 3090

Do this in /lab_notes/ before you ever run vLLM.

Model: Llama-3-8B (bf16, 8.03B params)
GPU:   RTX 3090, 24GB, ~700 GB/s effective HBM bandwidth
Regime: batch=1, decode (memory-bound)

The prediction:

Decode is memory-bound. Every token requires reading the full model weights + growing KV cache from HBM.

  • Model weights in bf16: 8.03B * 2 bytes = 16.06 GB

  • KV cache per token (Llama-3-8B: 32 layers, 8 KV heads, 128 head_dim, GQA): 2 * 32 * 8 * 128 * 2 bytes = 131 KB/token

  • At context length 2048: KV cache = 2048 * 131 KB = 268 MB (negligible next to weights)

  • Total HBM read per decode step ≈ 16.06 GB + 0.27 GB ≈ 16.33 GB

  • Predicted tok/s: 700 / 16.33 = ~43 tok/s

The measurement (what you’d actually see with vllm bench):

Roughly 55-70 tok/s depending on engine. Why higher than predicted?

  • Effective bandwidth on 3090 for well-tiled ops is closer to 750 GB/s, not 700

  • Weight-only quant (if you didn’t disable it) drops the read

  • FlashAttention decode kernels are quite tight

The lesson isn’t “I was wrong.” The lesson is: I predicted 43, measured 62, gap is +44%. Where does the 44% live? Now you actually understand your GPU.

Do this exercise. Then repeat it on a 4090 (~1000 GB/s), an A100 (~1900 GB/s), an H100 (~3350 GB/s). Suddenly you can predict throughput on hardware you’ve never touched.

Worked example 2: Max batch size on 24GB at 8k context

Model: Llama-3-8B bf16 on RTX 3090 (24GB VRAM)
Context: 8192 tokens
Question: What's the max batch size before OOM?

Arithmetic:

  • Model weights: 16.06 GB

  • CUDA runtime + PyTorch overhead: ~1.5 GB (empirical, benchmark this once and pin it)

  • Available for KV cache + activations: 24 - 16.06 - 1.5 = 6.44 GB

  • KV cache per token: 131 KB (from above)

  • KV cache per sequence at 8192 tokens: 8192 * 131 KB = 1.07 GB

  • Predicted max batch: 6.44 / 1.07 = 6 sequences

The measurement:

vLLM with --max-num-seqs 6 --max-model-len 8192 — does it fit? If yes, try 7. If OOM at 7, you predicted correctly.

If your prediction is off by 1-2, the gap is usually: paged-attention block granularity, activation memory during prefill, or CUDA graphs pre-allocating buffers. Each miss teaches you something specific.

The surprise_log.md

At the root of /lab_notes/, a single running log. Rules:

When your prediction misses by more than 30%, write an entry. No exceptions.

Template:

## 2026-08-08 — Softmax fusion speedup off by 20%... actually not a surprise, skip

## 2026-08-12 — Predicted 43 tok/s on Llama-3-8B/3090, measured 68 tok/s (+58%)
- Suspected cause: I used naive HBM bandwidth, not effective. Also underestimated
  KV cache read amortization for GQA (8 KV heads, not 32).
- Fix to mental model: for GQA models, KV read ~ (n_kv_heads / n_heads) x baseline.
- Corrected mental model: Llama-3-8B decode on 3090 ≈ 60-70 tok/s at ctx<4096.

## 2026-08-19 — Predicted 1.4x speedup from CUDA graphs, measured 2.3x (+64%)
- Suspected cause: I only accounted for launch overhead. Also removed CPU-side
  Python overhead in the sampling loop, which was bigger than I thought.
- Fix: CPU-side overhead in generation loops is 30-50% for small batches on
  fast GPUs. Graph capture kills that, not just kernel launches.

Read the surprise log every Sunday. 15 minutes. That’s your compounding tax. It’s the single highest-ROI reading you will do all week — because it’s the reading of your own mistakes, which are exactly calibrated to your current gaps.

By month 6 the log will have 40-60 entries. By month 13 it will be your single most valuable document. Nothing else — not any paper, not any repo, not any lecture — will teach you about your specific blind spots the way this file will.

The three-line check before every experiment

Sticky note on your monitor:

1. What number am I about to measure?
2. What do I expect that number to be, and why (with arithmetic)?
3. If the number surprises me, what will I have learned?

If you can’t answer all three in 60 seconds, you’re not ready to run the experiment. Go back and think.

The one-week test

One week in: your /lab_notes/ folder should have at least 3 files. One month in: at least 15 files. Six months in: at least 80 files and 15-25 surprise-log entries.

If you’re below those numbers, you are running the roadmap wrong. Not because the numbers are magical, but because they’re the natural rate of experiments if you’re actually doing the work. Below that = you’re passively reading, not actively building.

The lab notebook is the proof of work.