GPU MODE — the community

One of the highest-leverage things you can do this year is live inside the GPU MODE community. It is a Discord + YouTube + open-source ecosystem where the people who actually write frontier kernels (Meta, NVIDIA, Anthropic, PyTorch, vLLM, Hazy Research) show their work weekly. It is free. It has an active leaderboard. Join it.

Rebrand note: GPU MODE was called CUDA MODE until early 2024 — the rename reflects the community’s expansion to Triton, ROCm, and non-CUDA stacks.

What GPU MODE actually is

  • Weekly lecture (YouTube, streamed): ~1 hr talks by kernel authors. As of mid-2026, the lecture count is north of ~106.

  • Discord: channels for CUDA, Triton, ROCm, PyTorch internals, MLPerf, and specific kernel projects. Direct DMs with authors are common.

  • Leaderboards (KernelBot): pinned kernel problems (vec-add, matmul, sorting, softmax, matmul-fp8, MoE gate, attention variants). You submit a kernel; the bot benchmarks it on real GPUs; you get a rank. This is the fastest way to get feedback on whether your kernel is competitive.

  • Reading groups: recurring paper-discussion channels.

2026 direction — what you should know

GPU MODE’s 2026 focus (announced by Mark Saroufim, https://gpumode.com/news/gpumode-2026) is Kernel LLMs: post-training LLMs so they can write kernels that get merged into PyTorch and vLLM. Partners include Prime Intellect, Modal, Lambda, MIT. This means:

  • KernelBook (Sahan Paliskara, PyTorch) — dataset of kernels for training/eval

  • KernelBench (Simon Guo, Stanford) — the benchmark: 250 tasks + 20 aspirational HuggingFace tasks. Metric is fast_p = fraction of tasks where an LLM-generated kernel is both correct and ≥ p× the reference speed

  • KernelLLM (Zacharias Fisches, FAIR) — Meta’s kernel-generating model

  • KernelBot (Alex Zhang, Princeton) — the Discord evaluator

Even if you don’t work on the LLM side, the byproducts — leaderboard problems, curated datasets, tight feedback loops — are the highest-quality training environment for a human writing kernels.

Lectures to prioritize (as of July 2026)

Watch these in order. You don’t need to watch all ~106 lectures — target these first.

Foundations (weeks 1–2 of Phase 2):

  • Lecture 1 — Andreas Köpf, PMPP kickoff (aligns with 02_pmpp_path.md)

  • Lecture 3 — CUDA C++ intro

  • Lecture 5 — tiling, shared memory

  • Lecture 8 — CUDA Performance Checklist (Mark Saroufim) — this is the reference cheatsheet lecture

Kernel-writing (weeks 3–6):

  • Lecture 11 / 12 — Triton — Phil Tillet, William Malpica — canonical Triton walkthrough

  • Lecture 14 — CUTLASS overview

  • Lecture 15 — CUTLASS deep dive (Andreas Köpf)

  • Lecture on FlashAttention forward (search “FlashAttention Triton” in the playlist)

Frontier kernels & profiling (weeks 6–9):

Modern DSLs (weeks 8–9):

Tip: GPU MODE at GTC 2025 recap video is a fantastic 3-hour high-density crash course covering KernelBench, KernelBot, KernelBook, KernelLLM, ThunderKittens, CUTLASS. Watch it before you dive into individual lectures: https://www.youtube.com/watch?v=FtgXueoQkA0

KernelBench — what it is + how to use it

Source: github.com/ScalingIntelligence/KernelBench — 250 PyTorch modules across levels 1–4 (Level 1: single ops; Level 2: fused sequences; Level 3: full architectures; Level 4: aspirational HF tasks). Reference solutions are PyTorch modules; submitted solutions can be CUDA/Triton/etc.

Metric fast_p: fraction of tasks where your kernel is correct AND at least p× the PyTorch baseline. fast_1 = better than PyTorch baseline. fast_2 = 2× faster. The higher p, the harder.

How to use it as a human:

  1. Clone the repo, run the reference on your GPU to establish baselines.

  2. Pick a Level-1 op you understand (e.g., a specific reduction, a fused activation).

  3. Write your kernel (Triton or CUDA) with the same signature.

  4. Run the KernelBench harness locally to measure speedup.

  5. When you beat PyTorch by 1.5–2× and it’s correct: submit to KernelBot on Discord.

Even if you never submit, this is a honest feedback loop — you cannot bullshit “my kernel is fast.”

KernelBot leaderboard — how it works

  • Runs inside GPU MODE Discord. /leaderboard command lists open problems.

  • Submit a Python/CUDA/Triton file. Bot spins up a real GPU (H100/A100/etc.), runs correctness + benchmark.

  • Public ranking. Top submissions are studied and often become GPU MODE lecture content.

  • NVIDIA’s cuda.compute topped the leaderboard in Oct 2025 (https://developer.nvidia.com/blog/topping-the-gpu-mode-kernel-leaderboard-with-nvidia-cuda-compute), which gives you a public reference point for what “top score” means on problems like vec-add / matmul.

Entry strategy (concrete)

  • Week 1: join Discord. Post nothing. Read #introductions, #cuda, #triton.

  • Week 2–4: watch Lectures 1, 3, 5, 8. Do PMPP exercises in #study-group channels.

  • Week 5: pick one KernelBench Level-1 problem you understand. Write a CUDA version. Post benchmark in #kernel-golf.

  • Week 6: watch Lectures 11/12, port your kernel to Triton, compare.

  • Week 8: submit to KernelBot on a real leaderboard problem. Don’t aim to win; aim to land in the top 50%.

  • Ongoing: watch new lectures the week they drop.

Rules of engagement in the community

  • Ask specific questions. “Why is my matmul 40% of cuBLAS?” — attach ncu report, kernel code, hardware. Not “why is CUDA hard.”

  • Search before asking. Almost every beginner question has a pinned answer.

  • Contribute back. Write up your ncu findings. Fix a doc. Reproduce a paper.

  • Do not lurk forever. After you have working code, post it.

References