Rung 5 — The Mini Inference Engine¶
Aligned phase: Phase 4 capstone (engine internals) Ship by: v0.1 by end of M8, v1.0 by end of M11 Effort: 8 weeks, ~60–80 hrs Signal: high. This is the fulcrum of the entire ladder. Every rung before it prepares you to write this. Every rung after it references it.
The artifact spec¶
Build a ~800–2000 LOC Python inference engine that serves one small model family (Qwen 2.5 1.5B/3B/7B or Llama 3.2 1B/3B) via an OpenAI-compatible streaming API, with continuous batching, paged KV cache, prefix caching, and a benchmark harness that reproduces vllm bench serve output. Then benchmark it honestly against vLLM on the same hardware and publish the analysis.
Reference build: 05_engines/09_mini_engine_capstone.md. That file has the week-by-week plan. This file tells you how to turn the resulting repo into a portfolio artifact people will actually engage with.
Why this rung is the fulcrum¶
Rung 6 (first PR) becomes tractable because you have opened every abstraction in vLLM and hit the same design corners.
Rung 7 (reference architecture) is the writeup of what an engine like this is embedded in.
Rung 8 (sustained OSS) requires you already know an engine cold, and yours will be more forgiving to break than upstream.
Hiring managers scan for this specific repo. “Continuous batching + paged KV + streaming API” in an English-language README, with a public benchmark against vLLM, is the filter that separates people who talked about inference from people who built it.
Architecture diagram (put this in the README verbatim)¶
Copy that mermaid block into your README. Do not draw a PNG diagram — mermaid renders on GitHub and stays in sync when you edit.
The four design decisions to defend in prose¶
Every inference engine writeup that gets read is organized around a small number of decisions, not features. Write one section per decision. Each section: what the alternatives were, what you picked, one sentence of arithmetic justifying the pick, one measurement showing it works.
1. Paged KV with block_size = 16 (not contiguous, not block=1)¶
“Contiguous allocation forces worst-case reservation; block=1 pays a table-lookup per token. Block=16 (vLLM’s default) is the empirical sweet spot: fragmentation stays under 5% at realistic length distributions, and the block-table indirection cost is one L2 hit. Measured max concurrent sequences on Qwen 2.5 7B at 8k context on a single 24GB GPU: ~52 with paged vs ~14 with contiguous. That 3.7× is where the throughput comes from.”
2. Triton paged attention (not raw CUDA, not just FlashInfer as a black box)¶
“FlashInfer is the fast path in production. Triton is the pedagogical path — I can read and modify my own kernel. Ships with FlashInfer as the default backend for benchmark honesty; ships with my Triton paged-attention kernel behind a
--attn tritonflag so the code is reviewable and the numerics are inspectable. Kernel is ~60% of FlashInfer at batch 32; that’s the honest gap.”
3. FastAPI + asyncio (not ZMQ, not Ray)¶
“vLLM V1 uses ZMQ for the engine-process boundary because it needs to survive engine crashes. My engine is one process, one GPU; asyncio.Queue between the API layer and the scheduler loop is 50 lines instead of a ZMQ dependency, and the crash surface is not what I’m optimizing for. If a customer needs multi-process, vLLM is already the answer. This engine’s job is to teach and to benchmark.”
4. Prefix caching via hash-of-blocks dict (not radix tree)¶
“SGLang’s radix tree is cleaner for eviction and for shared-suffix scenarios. A dict of block-hash → physical-block-id is 30 lines and covers the same steady-state hit rate on multi-turn agent traces (measured within 3% of vLLM’s rate on a replayed Zoho-like agent trace). Radix tree is the v2 upgrade; I ship v1 first.”
One prescription per decision. No “options.” That is the voice. It is also the truth: you built this by picking, not by hedging.
The money shot: latency-throughput curves vs vLLM¶
The plot¶
X-axis: request rate (Poisson arrivals) at 1, 2, 4, 8, 16, 32 req/s. Y-axis: TTFT p95 (one plot) and ITL p95 (second plot). Two lines: vLLM, yours. Same hardware, same model, same seed, same ShareGPT trace.
Expected shape (be honest about the gap)¶
Rate 1–2 req/s: you should be within 1.5–2× of vLLM. Both are lightly loaded.
Rate 8 req/s: you should be within 2–3×. This is where vLLM’s CUDA graph capture pays off and yours doesn’t (unless you did the stretch item).
Rate 16–32 req/s: the gap widens to 3–5×. This is where FlashInfer’s paged attention, vLLM’s chunked prefill scheduling, and the multi-process engine architecture compound.
Publish where you lose. The credible writeup names each gap and points at the file in your repo where the future fix belongs. Example table in the writeup:
| Gap component | Contribution | Fix location |
|--------------------------|--------------|------------------------------------|
| Python scheduler overhead| ~25% | engine/scheduler.py — rewrite in C |
| No CUDA graph capture | ~30% | engine/executor.py |
| Triton attn vs FlashInfer| ~25% | kernels/paged_attn.py |
| No mixed prefill/decode | ~15% | engine/scheduler.py |
| Tokenizer/detok on Py | ~5% | api/tokenize.py |
That table is what makes reviewers stop scrolling. It says: I understand exactly where I lose, and I have a plan.
GitHub README structure (steal this outline)¶
# <engine-name>
One-line description. "A ~1500-LOC inference engine with continuous batching,
paged KV, prefix caching, and OpenAI-compatible streaming. Built to learn;
benchmarked against vLLM to keep honest."
[Hero plot: TTFT/ITL vs vLLM] [Badge: benchmarks reproducible]
## Quickstart
```bash
pip install -e .
python -m myengine.serve --model Qwen/Qwen2.5-3B-Instruct --port 8000
# in another terminal
curl -N http://localhost:8000/v1/chat/completions -d '{...}'
Architecture¶
[mermaid diagram from above]
Benchmarks vs vLLM¶
[table + plot + honest gap analysis]
Design decisions¶
[the four sections from above]
What’s here / what’s not¶
[MVP checklist + explicit non-goals]
Reproducing the numbers¶
[commands to regenerate every plot]
References¶
[nano-vllm, flex-nano-vllm, vLLM, SGLang, FlashInfer]
The order matters: **plot first, quickstart second, architecture third.** People decide whether to keep reading in 10 seconds. The plot is the hook.
---
## The blog post structure
Different from the README. The blog is narrative; the README is reference.
1. **The hook (2 paragraphs).** "I built a 1500-line inference engine to see how close to vLLM I could get on one GPU. Answer: 3–5× off at rate 16 req/s. Here's every place I lose."
2. **The three-numbers primer (1 paragraph).** Remind the reader: decode is memory-bound, prefill is compute-bound, the whole scheduling literature is choosing operating points on the latency-throughput curve. Assume they know it; you're just planting the vocabulary.
3. **The architecture (mermaid + 300 words).**
4. **Continuous batching — what actually changed my throughput 10×.** One code block: the scheduler loop. Show the before (static batching) and after (iteration-level).
5. **Paged KV — why block_size=16 is not arbitrary.** Show the arithmetic on fragmentation.
6. **Prefix caching — the agent workload win.** Replay a two-turn chat, measure TTFT collapse, show the hit rate log.
7. **The vLLM benchmark and the honest gap.** Plots, gap-attribution table.
8. **What I'd do next.** Three items, ranked by ROI. This is your credibility signal — you're not done, you know what's next.
9. **How to reproduce.** Every command. This is what earns the stars.
**Target length: 2500–4000 words.** Longer than the matmul post, shorter than the FA2 post. The topic warrants depth; the depth is in the code, not the prose.
---
## Reference minimal engines (study, do not copy)
1. **GeeeekExplorer/nano-vllm** — https://github.com/GeeeekExplorer/nano-vllm. ~1000 lines, mirrors vLLM's API, TP + CUDA graphs + prefix caching. **Read this before you start.** Their bench claim on RTX 4070 (~1400 tok/s vs vLLM ~1360) is worth verifying yourself. Steal the file layout. Do NOT steal the implementation.
2. **changjonathanc/flex-nano-vllm** — https://github.com/changjonathanc/flex-nano-vllm. FlexAttention-based. If writing your own paged attention kernel is a bridge too far in 8 weeks, this is the shortcut. Blog: "vLLM flex attention from scratch."
3. **karpathy/nanoGPT** — reference for clean PyTorch model code, not for serving.
4. **vLLM V1 source** — read `vllm/v1/engine/*.py` and `vllm/v1/core/*.py` when you hit a design question. `10_vllm_source_map.md` in this roadmap has the exact file paths.
5. **SGLang `python/sglang/srt/mem_cache/radix_cache.py`** — the cleanest radix-tree prefix-cache implementation in any open engine. Read it in Week 5.
**The rule:** you may read any of these; you may not paste any of them. If you cannot type a subsystem from memory after reading, read again. The learning is in the retyping.
---
## Where to post
1. **GitHub:** `github.com/<you>/<engine-name>`. Pin. **Star it yourself and get 3 friends to star it in the first 24 hours** — GitHub's discovery algorithm cares about the first-day slope.
2. **Blog:** cross-post. Title: **"A 1500-LOC inference engine, benchmarked honestly against vLLM."** The word "honestly" is the hook — nobody in the field says it.
3. **r/LocalLLaMA:** submit. Sunday afternoon IST. Lead with the honest-gap table.
4. **HN Show HN:** worth trying. HN loves "small-vs-big" honest benchmarks. Post Monday morning US time.
5. **GPU MODE Discord `#vllm` or `#inference`:** share once. Ask for review from someone who has actually contributed to vLLM.
6. **vLLM Discord / Slack:** share in `#general` or the dev channel. If Woosuk Kwon or Simon Mo comment, thank them and take the feedback seriously.
7. **X/Twitter:** thread. Tag `@vllm_project @lmsysorg @woosuk_k @simon_mo_`. Only if the numbers actually hold up.
8. **LinkedIn:** yes, now. This is the artifact that unlocks LinkedIn as a signal channel. Post the hero plot and one paragraph.
---
## Success signals (measure at 4 weeks)
- [ ] **100+ GitHub stars** organically. This is the threshold where recruiters start finding the repo.
- [ ] **Shoutout in GPU MODE / vLLM meetup / SGLang office hours** as a reference minimal engine. Nano-vllm got this; you can too if the code is clean and the writeup is honest.
- [ ] **1 issue or PR from an outside contributor.** This means someone read your code deeply enough to have an opinion. That is a very high signal.
- [ ] **Cited in a follow-up blog post by someone else.** Not required, but if it happens ("like nano-vllm and <you>/<engine>") you have arrived.
- [ ] **DM or email from a hiring manager referencing this repo specifically.** This is what the rung is for.
If ≥2 of 5 hit, the artifact is working. If 0 hit after 6 weeks, the writeup is the gap — not the code. Rewrite the top of the README.
---
## The Zoho angle (natural, not forced)
One section of the blog: **"What this taught me about prefix caching for agent loops."**
> "At Zoho I write agentic harnesses — multi-turn tool-calling loops where the same system prompt + conversation history gets re-sent every turn. Building the prefix cache made me measure hit rates on a replayed agent trace: **83% on Turn 2, 91% on Turn 3, 94% on Turn 4+.** That number is the single biggest inference-cost win available to any team running an agent. It is also the thing you cannot appreciate until you have implemented the cache yourself."
One paragraph. No customer detail. The workload *shape* is what makes it credible. That paragraph turns the engine from "student project" into "practitioner-authored."
---
## Acceptance criteria (the exit gate before publishing)
1. Engine sustains **≥50 concurrent sequences** with continuous batching + paged KV on Qwen 2.5 7B at 8k context on a single 24GB GPU.
2. Streaming SSE output is byte-identical to the `openai` Python client's parsing.
3. On the ShareGPT trace at 8 req/s, you are within **3×** of vLLM's throughput. Within 2× is excellent. Within 1.5× is a talk at GPU MODE.
4. Prefix cache hit rate on a repeated-prefix trace matches vLLM's within **5%**.
5. `bench_serve.py` output format matches `vllm bench serve` — side-by-side comparison is trivial.
6. Public repo + benchmark writeup + reproducible commands.
7. You can walk someone through the request lifecycle from memory: arrive → tokenize → schedule (with block allocation + prefix hash check) → prefill → decode iterations → detokenize + stream → free blocks. No notes.
If ≥6 of 7 hit, ship.
---
## What signals it sends
- "I have built the primitives, not just used them."
- "I understand vLLM's design well enough to explain each of my divergences from it."
- "I can profile and benchmark a serving system honestly."
- "I know where I lose and I have a plan for closing each gap."
- "I can ship 1500 lines of Python that does something real."
This is the artifact that changes the phone screen. Before this rung: "tell me about serving." After this rung: "walk me through your engine's scheduler." The study posture flips.
---
## Common mistakes
1. **Scope creep to "vLLM but better."** You will lose to vLLM on every axis. That is fine. The point is the *build*, not the win. Cap features at MVP + one polish item.
2. **Static batching "just for the baseline" that becomes the ship.** Do not publish static batching as v1. Week 3 is when continuous batching lands; do not slip.
3. **Custom Triton attention before FlashInfer works.** Ship FlashInfer as the default backend. Add Triton as an optional flag once the engine is end-to-end.
4. **No benchmark harness.** The engine without `bench_serve.py` is unpublishable. Do not defer the harness.
5. **Hiding the gap.** Reporting "comparable to vLLM" when you are 3× off is the credibility killer. The 3× is fine; the hiding is not.
6. **Publishing without the mermaid diagram.** The visual is the fastest way for a reviewer to decide whether to keep reading. Do not skip it.
7. **Waiting for v1.0 to publish.** Ship v0.1 at end of M8 on this repo publicly, even if half the polish items are still TODO. Momentum > polish.
---
## Success criteria checklist
- [ ] Continuous batching working, verified against static baseline
- [ ] Paged KV cache with block_size = 16 and prefix cache
- [ ] OpenAI-compatible `/v1/chat/completions` with SSE streaming
- [ ] Benchmark harness matching `vllm bench serve` output
- [ ] Latency-throughput plot vs vLLM at 6 request rates
- [ ] Gap-attribution table with 5+ rows
- [ ] Mermaid architecture diagram in README
- [ ] Blog post published on personal domain
- [ ] Zoho-flavored prefix-caching paragraph in the blog
- [ ] Repo builds and benchmarks reproduce from a clean env in ≤2 hours
If ≥8 of 10 hit, you are ready for rung 6.
---
## Next step
**On the Monday after you publish this, you** open the vLLM issue tracker, filter by `good first issue` + `help wanted`, and find one issue that touches a subsystem you now understand cold from your own build. Then you write a comment on that issue asking one specific implementation question. That comment is the seed of your first PR. Rung 5 exists to make rung 6 tractable; do not delay the transition.
**The ladder is the CV. Every rung is public. Every rung compounds.**