06 — Failure Modes¶

Seven ways this roadmap dies. Each one has a symptom, a diagnostic, and a remedy. Print this file. Read it whenever you feel stuck.

The two originals (from the seed roadmap)¶

These are the ones that will actually kill 90% of attempts. The other five are supporting cast.

Failure mode 1 — Tutorial-hopping¶

Symptom: You’ve been “studying” for 6 weeks. You’ve watched 40 hours of lectures, cloned 12 repos, bookmarked 200 blog posts. Your GitHub has no new commits. Your /lab_notes/ folder has 3 files, none from the last 10 days. When someone asks what you built, you list what you read.

Diagnostic: Look at git. If the ratio of hours consumed to lines shipped is worse than 3:1 for the last two weeks, you’re tutorial-hopping.

Remedy:

Close every tab. Every one.
Open the current phase’s exit criterion. Read it.
Start the current sprint’s plan (00_plan.md). Set a deliverable that’s shippable in 2 weeks.
Ban new resources for the sprint. No new tutorials, no new lectures, no new bookmarks. Only what you already have.
If you don’t know enough to ship, ship the smallest version. Then learn what’s missing to make version 2.

Prevention: The exit criteria and the 2-week sprint deliverables exist exactly to prevent this. Trust the scaffolding. When in doubt: ship the current phase’s project.

Failure mode 2 — Premature depth¶

Symptom: You’ve spent 3 weeks on CUTLASS in month 2. You’re reading the Ampere whitepaper cover-to-cover before you’ve served your first model. You want to “really understand” register allocation before you write a Triton kernel. You bought Hennessy & Patterson and you’re on chapter 4.

Diagnostic: Ask “can I currently serve a real model to a real HTTP endpoint?” If no, and you’re in a rabbit hole below the abstraction level you need for that answer, you’re in premature depth.

Remedy:

Bookmark the rabbit hole. It’s not wrong; it’s just early.
Go up the stack. What phase are you actually in? What’s the exit criterion?
Deliver the current phase’s artifact using off-the-shelf tools. You’ll come back to CUTLASS in phase 3 or 4, better prepared and with actual context for what problems it solves.

Prevention: The phase ordering is the immune system. Foundations → transformers → kernels → attention → engines → quant → distributed → production. Every phase’s abstractions rest on the phase below and are used by the phase above. If you’re four floors down when the sprint is on floor 6, come back up.

The five additional modes (you’ll hit some of these too)¶

Failure mode 3 — Analysis paralysis (reading vs building)¶

Symptom: You’ve re-read the PagedAttention paper 4 times. You still don’t feel “ready” to try it. You keep telling yourself you need to understand it “properly” before you touch code. Meanwhile: three weeks, no code.

This is the introvert-engineer version of tutorial-hopping. It masquerades as diligence.

Diagnostic: If you can articulate the paper’s core idea in 3 sentences but haven’t written a single line of code touching the concept, you’re paralyzed, not preparing.

Remedy:

Set a 90-minute timer. Write the worst possible implementation of the concept in that time. It will be wrong. That’s the point.
Then diff your wrong implementation against the reference. The diff is the lesson. That diff will teach you more in 30 minutes than 4 more reads of the paper.
Rule: understanding follows action, not the other way around, for anything involving code.

Prevention: the lab notebook template starts with hypothesis + predicted number. That forces you into action before you’re “ready.” Trust it.

Failure mode 4 — Benchmark theater¶

Symptom: Pretty charts. Colorful plots. Log-scale axes chosen to make the line look dramatic. Numbers reported without workload description. “3.2x speedup!” without saying compared to what, on what hardware, at what quantization. Screenshots of nvitop posted as if that’s a measurement.

Diagnostic: Look at any benchmark you produced last week. Can a reviewer at Neural Magic or LMSYS take that repo, run one command, and get your number within noise? If no, you’re doing benchmark theater.

Remedy:

Re-read 03_benchmark_hygiene.md.
Kill the chart. Publish the JSON. Publish the script. Publish the environment.
If it doesn’t reproduce cleanly, delete the post. Better to have no benchmark than a bad one.

Prevention: the pre-publish checklist in 03_benchmark_hygiene.md. Tick every box.

Failure mode 5 — Tool-chasing¶

Symptom: Week 3, you’re learning CUDA. Week 5, someone tweets about Triton, and you switch. Week 8, Mojo drops a blog post; you consider learning it. Week 10, someone posts about ThunderKittens; you clone it. Week 12, you actually know how to write kernels in none of them.

Diagnostic: How many DSLs / libraries / frameworks did you seriously touch in the last quarter? If it’s more than 2, you’re chasing.

Remedy:

Pick one and finish the phase with it. For kernels: Triton first (highest leverage for your Python fluency), then CUDA (for depth), then optionally CUTLASS/Thunderkittens/tile-lang later.
New tool releases: bookmark, don’t switch. Add them to /lab_notes/tool_bookmarks.md with one line about what problem it solves.
Re-visit that list once per phase transition, not weekly.

Prevention: the phase resources in the roadmap are already opinionated. Follow them. Deviate only after you’ve completed a phase with the recommended stack.

Failure mode 6 — Isolation¶

Symptom: Month 4. You’ve done real work. Your /lab_notes/ is thick. You have 20 commits in a private repo. You’ve never posted anything. You haven’t spoken to another inference engineer. You’re doing all of this alone.

This is the failure mode that quietly kills the most people, because it looks like discipline. It’s not. It’s a lack of feedback.

Diagnostic: Has any person you’ve never met given you feedback on your work in the last month? If no, you’re isolated.

Remedy:

Post one thing this week. Even rough. Even small. A Reddit comment with a benchmark number. A tweet with a plot. A GitHub issue with a repro. Break the seal.
Join GPU MODE Discord this week (see 10_communities/). Read for a few days. Then post in #introductions.
Attend one virtual community event this month (GPU MODE reading group, vLLM meetup, LMSYS talk). Just show up in the audience.
Rule: every phase must produce at least one public artifact. No exceptions. This forces the seal to break.

Prevention: the 05_teach_to_learn.md cadence + 10_communities/ file. If you’re doing those, you’re not isolated.

Failure mode 7 — Impostor stall¶

Symptom: You read a vLLM PR from a 22-year-old MIT grad and feel a wave of “I’ll never catch up.” You look at Tri Dao’s CV. You spiral. You close the laptop. You don’t open it again for a week. When you do, you don’t retro. You just quietly slip a sprint.

Diagnostic: Are your retros starting to sound like apologies? “I didn’t get much done this week because…” Do you find reasons not to publish? Are you comparing yourself to people 5 years ahead and drawing the conclusion that you shouldn’t try?

This is not laziness. This is a specific failure mode with a specific remedy.

Remedy:

Read 07_motivation_sustainment.md — the whole thing.
Talk to a human. Not a tech human. A friend. See 08_health_burnout.md.
Radically shrink the next sprint. If you were going to build a full engine, ship a 200-line prefill-only version. Ship something. Small wins re-boot the identity.
Read your own /lab_notes/ from month 1. You’ve come farther than you remember.
Compare yourself to past you, not to Tri Dao. Tri Dao’s benchmark is Tri Dao. Yours is yours.

Prevention: the retro cadence + community engagement + the identity work in 07. You don’t prevent impostor thoughts — you prevent impostor stalls, by having the scaffolding to keep moving through them.

The universal tie-breaker¶

When you’re stuck between two paths and can’t decide, apply this in order:

Am I in one of the seven failure modes right now? If yes, apply that remedy.
What is the current phase’s exit criterion? Do the smallest next thing that moves you toward it.
Ship the current phase’s project. When in doubt, always this.

That’s the whole decision tree. Keep it simple.

The weekly self-check¶

Sunday evening, before writing the retro, ask three questions:

Am I tutorial-hopping? (Ratio: hours read vs lines shipped this week)
Am I chasing depth I don’t need yet? (Am I below the abstraction level of my current phase?)
Am I isolating? (Have I engaged with any other human on this work this week?)

Honest answers, in writing. If any is a red flag, address it in next sprint’s 00_plan.md.

The failure modes are not moral failures. They’re the natural shape of the terrain. You will hit them. The people who finish the 13 months are the ones who recognize the mode, name it, and apply the remedy. That’s it. That’s the difference.