The PMPP Path

Book: Programming Massively Parallel Processors: A Hands-on Approach, Hwu, Kirk, El Hajj — 4th edition (2022). This is the standard undergraduate/graduate CUDA textbook and still the fastest way from “I know what a GPU is” to “I can write CUDA that isn’t embarrassing.”

Note (July 2026): A 5th edition (ISBN 978-0-443-33245-4) is now out on Elsevier (https://educate.elsevier.com/book/details/9780443332454). It updates for Hopper/tensor cores and adds chapters on modern kernel patterns. However, the 4th ed has the deep bench of community solutions (e.g. github.com/tugot17/pmpp walkthroughs, GPU MODE Lecture 1 covers ch. 1–3 of 4th ed) and is the version referenced in nearly every reading list. Use the 4th ed as primary; skim the 5th ed’s new chapters at the end.

Why PMPP and not just tutorials

Tutorials teach you syntax. PMPP teaches you the why: why coalescing is a property of the addressing pattern, why bank conflicts are a property of stride, why prefix-sum has a specific tree shape. These are the exact abstractions you will reason with when you look at a Triton kernel or CUTLASS gemm and need to explain why it is fast.

Cadence

• 10 weeks, ~1 chapter/week with exercises. Skip chapters marked skip below — they don’t buy you inference muscle.

• Ship one code artifact per chapter. Push to a pmpp/ repo. Do not merely read.

Chapter-by-chapter map (4th ed)

Ch	Topic	Priority for inference	What to actually do
1	Intro	Skim	Read once.
2	Heterogeneous data-parallel computing	Read	Vector add, cudaMalloc / cudaMemcpy reflex.
3	Multidim grids & data	Read	2D grid, image blur or transpose.
4	Compute architecture & scheduling	Core	Warps, occupancy, divergence. Write a divergent-vs-non-divergent kernel and time it.
5	Memory architecture & data locality	Core	Tiled matmul (naive → shared memory). This is the SGEMM warm-up.
6	Performance considerations	Core	Coalescing, corner turning. Verify with Nsight.
7	Convolution	Skip-ish	Skim; you’re not writing conv kernels for LLMs.
8	Stencil	Skip	Skip.
9	Parallel histogram	Optional	Atomics + privatization mental model.
10	Reduction	Core	Log-tree reduction, tree balancing. Prereq for softmax.
11	Prefix sum (scan)	Read	Kogge-Stone / Brent-Kung. Underpins many kernels.
12	Merge	Skip	Skip.
13	Sorting	Skip	Skip.
14	Sparse matrix	Optional	If you’ll touch MoE routing or sparse attention, worth skimming.
15	Graph traversal	Skip	Skip.
16	Deep learning	Read	Foundational, dated. Read for the framing; real deep-learning kernels live in PMPP-adjacent papers.
17	Iterative MRI	Skip	Skip.
18	Electrostatic potential map	Skip	Skip.
19	Parallel programming & CUDA parallelism	Read	Streams, events. Prereq for CUDA graphs.
20	Programming a heterogeneous computing cluster	Skim	You’ll get distributed material in Phase 7.
21	CUDA dynamic parallelism	Skip	Not relevant for inference.
22	Advanced practices & future evolution	Read	Tensor cores intro (dated but useful).
23	Conclusion	Skim	Fine.

Net: Chapters 2, 3, 4, 5, 6, 10, 11, 16, 19, 22 are the inference spine. That’s 10 chapters ≈ 10 weeks.

Exercises worth doing (concrete list)

Ship each of these as separate .cu files with a Makefile and a bench.py that verifies correctness against torch and prints achieved GB/s or TFLOPs.

1. Ch. 2 — vector add with pinned host memory and correctness check vs numpy.

2. Ch. 3 — matrix transpose — write naive, then a tiled+padded version. Measure GB/s vs cuBLAS / torch.transpose. Learn about bank conflicts here, not later.

3. Ch. 4 — divergence experiment — same kernel with vs without if (tid % 2) branching, measure the throughput gap.

4. Ch. 5 — tiled SGEMM (32×32 tile) — this is the launching pad for 04_sgemm_ladder.md. Get within ~15% of a naive baseline first, know why it’s slow.

5. Ch. 6 — coalescing — write two versions of a kernel that touch the same memory in coalesced vs strided patterns. Report the delta.

6. Ch. 10 — parallel reduction — implement all 7 versions from Mark Harris’s classic slides (reduction/nvidia). Log-2 timings. This is prep for softmax.

7. Ch. 11 — Kogge-Stone scan — needed as a mental building block for online algorithms.

8. Ch. 19 — streams demo — overlap host↔device copy with compute using CUDA streams. Sets up CUDA graphs later.

Each exercise should end with an Nsight Compute report attached. See 07_profiling_mastery.md for how to gather them.

GPU MODE Lecture 1 tie-in

GPU MODE Lecture 1 (Andreas Köpf) is a companion to PMPP chapters 1–3. Watch it before starting: https://www.youtube.com/@GPUMODE — it will save you 4 hours of setup pain. Subsequent GPU MODE lectures pick up specific PMPP chapters piecemeal (see 03_gpu_mode.md for the map).

What to do if you don’t have a GPU

• Colab T4 / L4 free tier is fine for chapters 1–11.

• Lightning AI Studios free credits, ~ hour of H100 time.

• Modal and Lambda offer per-second GPU billing; you’ll spend < $30 to run every PMPP exercise on H100.

• RTX 4090 / RTX 5090 local: ideal but not required. Enough to complete SGEMM and Triton work.

Common failure modes

• Reading without writing. Every chapter must produce code. If you can’t reproduce the reduction ladder from memory, you did not learn it.

• Skipping Nsight until the SGEMM chapter. Wrong. Profile the vector add. Profile the transpose. Build the habit early.

• Optimizing without a baseline. Every kernel starts with the naive version and a measurement. Speedup without a baseline is theater.

References

• PMPP 4th ed: https://www.amazon.com/dp/0323912311 (Elsevier / Morgan Kaufmann)

• PMPP 5th ed (new, 2026): https://educate.elsevier.com/book/details/9780443332454

• Community solutions (4th ed): https://github.com/tugot17/pmpp

• GPU MODE Lecture 1 (PMPP kickoff): https://www.youtube.com/@GPUMODE

• Mark Harris reduction slides (classic): https://developer.download.nvidia.com/assets/cuda/files/reduction.pdf