Phase 2 — Projects (with acceptance criteria)

Rule: ship code, not notes. Everything in Phase 2 crystallizes into these artifacts. If a project doesn’t produce a repo directory with (a) working code, (b) a benchmark script, (c) a reproducible Nsight report, and (d) a README with numbers — it doesn’t count.

Target hardware: any modern NVIDIA GPU (RTX 4090 ideal for local, L4/H100 on cloud). If you’re on Colab T4, some Blackwell-specific projects are aspirational; do the rest.

Suggested repo layout: one top-level phase2/ folder with one subfolder per project below. Each project has a README.md, bench.py, report/ (with .ncu-rep files), and the kernel source.

P2.1 — PMPP capstone: reduction ladder

Goal: implement Mark Harris’s 7-version parallel reduction (sum) from PMPP Ch. 10 / Harris slides. Each version incrementally applies one optimization.

Deliverables:

  • reduce_v1_naive.cu through reduce_v7_final.cu

  • Benchmark script that runs all 7 versions on N=2^24 fp32, prints achieved GB/s

  • Nsight Compute report per version

Acceptance criteria:

  • V7 achieves ≥ 75% of HBM peak on your GPU.

  • Correctness against numpy.sum at fp32 (tolerance 1e-3 relative).

  • You can explain each version’s optimization out loud without notes.

Time: 3–4 days.

P2.2 — SGEMM ladder (the big one)

Goal: walk Simon Boehm’s 10-step SGEMM ladder yourself. See 04_sgemm_ladder.md for the step list.

Deliverables:

  • sgemm_00_cublas_reference.cu through sgemm_09_double_buffer.cu (skip step 10 for now)

  • Benchmark script sweeping M=N=K in {512, 1024, 2048, 4096} on fp32

  • Nsight Compute report per step (at M=N=K=2048)

  • Speedup table (this step vs previous, and this step vs cuBLAS as %)

Acceptance criteria:

  • Final kernel hits ≥ 70% of cuBLAS on M=N=K=2048 fp32 on your target GPU. ≥ 80% is target quality. 90%+ is exceptional.

  • Non-monotonic regressions are debugged (i.e., every step is at least as fast as the prior).

  • You can reproduce step 8 from memory.

Time: ~2 weeks. Do not rush this.

P2.3 — Triton warm-up trio

Goal: three Triton kernels that will be your building blocks in Phase 3.

P2.3a — Fused softmax (Triton)

Deliverable: fused_softmax.py implementing row-wise softmax on (B, N) fp32 and bf16 tensors. Autotune with 4+ configs. Acceptance: ≥ 2× faster than torch.softmax(x, dim=-1) on (16384, 8192) bf16. Correct within atol=1e-2 vs torch.

P2.3b — Fused RMSNorm (Triton)

Deliverable: fused_rmsnorm.py implementing y = weight * x / rms(x) with rms(x) = sqrt(mean(x^2) + eps). Compute mean-of-squares in fp32 even if input is bf16. Acceptance: ≥ 3× faster than a naive x / x.pow(2).mean(-1, keepdim=True).sqrt() * weight. Correct within bf16 tolerances.

P2.3c — Fused SwiGLU forward (Triton)

Deliverable: fused_swiglu.py implementing y = silu(x_gate) * x_up where the input is a single tensor with the gate/up projections concatenated (real Llama layout). Acceptance: ≥ 2× faster than the equivalent torch.nn.functional.silu(a) * b. Correct bf16.

Combined time: 1 week.

P2.4 — Roofline analysis notebook

Goal: for one shape of one kernel (e.g., your final SGEMM at M=N=K=2048), compute by hand: arithmetic intensity, ridge point, expected roofline placement. Compare to Nsight’s roofline plot.

Deliverables:

  • roofline_analysis.ipynb (or .md with math)

  • Table: (kernel, shape, HBM bytes moved, FLOPs, intensity, achieved TFLOPs/s, expected TFLOPs/s from roofline, achieved / expected %)

  • Screenshots of Nsight’s roofline plot

Acceptance criteria:

  • Your hand-computed intensity is within 20% of Nsight’s measured value.

  • Your predicted roofline placement matches Nsight’s plot.

  • You extend this to at least 3 kernels (SGEMM, softmax, RMSNorm).

Time: 2 days.

P2.5 — CUTLASS example modification

Goal: touch CUTLASS at least once. Compile and modify one example.

Deliverables:

  • Fork of CUTLASS with a single-file modification to examples/48_hopper_warp_specialized_gemm (or a similar Ampere example if no Hopper access): change block tile, num stages, or pingpong config.

  • Before/after benchmark output.

  • Written note explaining what you changed and the perf delta.

Acceptance criteria:

  • It compiles and runs.

  • You can articulate what your change did to the pipelining.

Time: 2 days. Do not spend more.

P2.6 — KernelBench Level-1 submission

Goal: land a submission on the GPU MODE KernelBot leaderboard.

Deliverables:

  • Chosen problem (any Level-1) + your kernel (Triton or CUDA)

  • Local benchmark showing ≥ 1.2× speedup over the PyTorch baseline

  • Submission via /leaderboard in the GPU MODE Discord

  • Your rank at submission time

Acceptance criteria:

  • Correctness: passes KernelBench’s harness.

  • Speedup: > 1× vs PyTorch reference on the target GPU.

  • Bonus: top 50%. Aspirational: top 10.

Time: 3 days.

P2.7 — Awareness: AMD port

Goal: port one of your Triton kernels (fused softmax or RMSNorm) to run on MI300X. Just run it, tune it minimally, benchmark.

Deliverables:

  • Same Triton file running on MI300X (Modal / Runpod / Hot Aisle)

  • Autotune re-run (config sweep will differ)

  • Bench comparison: your kernel on H100 vs MI300X vs baseline on each

Acceptance criteria:

  • Runs correctly on MI300X.

  • You can articulate one thing that changed in the autotune winner.

Time: 1 day.

P2.8 — The profiling artifact

Goal: one exemplary Nsight report + write-up that you can show in an study.

Deliverables:

  • Pick your best kernel from P2.2 or P2.3.

  • Produce a one-page technical report: SOL, memory workload, warp stall analysis, roofline, and the top-3 optimizations you applied with before/after Nsight metrics.

  • The .ncu-rep file attached.

Acceptance criteria:

  • You could hand this to a staff engineer and defend every number in it.

  • The write-up is dense, factual, no filler.

Time: 1 day.

Phase 2 exit study (self)

Before moving to Phase 3, sit down and truthfully answer:

  1. Can I compute the H100 BF16 ridge point from memory? (Yes/No)

  2. Can I draw the H100 memory hierarchy with bandwidths? (Yes/No)

  3. Is my SGEMM ≥ 70% of cuBLAS at M=N=K=2048? (Speedup number: ___)

  4. Can I write a fused softmax in Triton without looking at any tutorial? (Yes/No)

  5. Can I produce and read an Nsight Compute report? (Yes/No)

  6. Do I have a KernelBot submission? (Rank: ___)

  7. Have I read Horace He’s Brrr post? Can I recite the compute/memory/overhead trichotomy? (Yes/No)

  8. Do I have an opinion on when to use Triton vs CUTLASS? (Yes/No)

If any answer is “No” or the SGEMM number is below 70%, do not proceed to Phase 3 yet. Fix the gap.

Rollup: the physical artifacts you own at end of Phase 2

  • phase2/reductions/ — 7 kernels, Nsight reports, bench data

  • phase2/sgemm/ — 10 kernels, Nsight reports, bench data, cuBLAS reference

  • phase2/triton_warmup/ — fused softmax, RMSNorm, SwiGLU

  • phase2/roofline/ — analysis notebook with 3 kernels placed

  • phase2/cutlass_mod/ — one modified example + bench

  • phase2/kernelbench_sub/ — leaderboard submission

  • phase2/amd_port/ — one kernel ported to MI300X

  • phase2/report/ — the one-page Nsight technical report

Total: ~8 subdirs, ~30 kernels, ~30 Nsight reports, ~1 leaderboard entry. This is real portfolio evidence.