09 — Mini Inference Engine (The Capstone)

This is the single most important deliverable in the 13-month roadmap. Every phase before it is instrumental; every phase after it is amplification. If you finish only one project publicly, finish this one.

Why this project matters

Reading vLLM source is instructive. Reading it after having built a toy that solved the same problems is transformative — you know why each abstraction exists because you tried the alternatives and they broke. That’s what separates “familiar with vLLM” from “can debug vLLM at 3am.”

For the job market: a working continuous-batched, paged-KV, streaming OpenAI-compatible engine — even a 2000-line Python one — is a stronger signal than any certificate. Combine it with the honest benchmark write-up vs vLLM and you have a portfolio piece that opens doors at anyone hiring inference engineers.

For you specifically: your Zoho async/harness experience gives you a huge head start on the server + scheduler loop. The kernels and paged KV are the new muscle.

Scope: what you’re building

A serving engine for one model family (recommend Qwen 2.5 1.5B/3B/7B or Llama 3.2 1B/3B — small enough to iterate fast, real enough to matter) with:

Must-have (MVP):

  1. Async HTTP server (FastAPI/uvicorn) with OpenAI-compatible /v1/chat/completions and /v1/completions, streaming SSE.

  2. A single-GPU model executor: load safetensors, run forward, decode.

  3. Continuous batching scheduler: iteration-level, waiting queue, decode-first.

  4. Paged KV cache with block manager: allocate/free, prefix reuse via hashing.

  5. Sampling: greedy, temperature, top-k, top-p. Logit-processor interface.

  6. Benchmark harness that reproduces vllm bench serve’s output format (Poisson arrivals, ShareGPT distribution, p50/p95/p99 for TTFT/ITL).

Stretch (do at least one):

  • Chunked prefill (token budget per step)

  • CUDA graphs for decode

  • N-gram / prompt-lookup speculative decoding

  • Structured output via XGrammar (it’s a library — you just wire the logit processor)

  • FlashInfer or your own Triton paged-attention kernel

Explicitly out of scope for MVP: tensor parallelism, multi-node, quantization beyond fp16/bf16, LoRA hotswap, tool-calling parsers. Add these only after MVP ships.

Architecture — mirror vLLM V1 but simpler

┌────────────────────────────────────────────────────────────┐
│  API server process (asyncio, FastAPI)                     │
│  - tokenize, format chat template                          │
│  - stream chunks back to client                            │
│  - talks to engine via asyncio.Queue (not ZMQ — simpler)   │
└──────────────────────┬─────────────────────────────────────┘
                       │  (RequestInput / RequestOutput)
┌──────────────────────▼─────────────────────────────────────┐
│  Engine core loop (single process, run in thread)          │
│  ┌────────────────────────────────────────────────────┐    │
│  │ Scheduler                                          │    │
│  │  - waiting queue (new requests)                    │    │
│  │  - running list (in-flight sequences)              │    │
│  │  - schedule() → (batch, positions, block_tables)   │    │
│  └────────────────────────────────────────────────────┘    │
│  ┌────────────────────────────────────────────────────┐    │
│  │ KV cache manager                                   │    │
│  │  - block pool (physical blocks, free list)         │    │
│  │  - prefix hash → block IDs (prefix cache)          │    │
│  │  - block tables per sequence                       │    │
│  └────────────────────────────────────────────────────┘    │
│  ┌────────────────────────────────────────────────────┐    │
│  │ Model executor                                     │    │
│  │  - forward(input_ids, positions, block_tables)     │    │
│  │  - paged attention kernel (Triton or FlashInfer)   │    │
│  │  - sampler                                         │    │
│  └────────────────────────────────────────────────────┘    │
└────────────────────────────────────────────────────────────┘

Single-process for the MVP. Multi-process (vLLM V1 style) is an easy refactor later if you want.

Week-by-week build plan (8 weeks, ~10 hrs/week)

Assumes you finished Phases 1–3. Budget 60–80 hours total.

Week 1 — Foundation

  • Load a Qwen 2.5 1.5B checkpoint from HuggingFace safetensors into your PyTorch inference-only model (built in Phase 1).

  • Verify greedy output matches HF transformers on 5 prompts.

  • Write a Request / Sequence dataclass: req_id, prompt_ids, output_ids, sampling_params, state (WAITING|RUNNING|FINISHED).

  • Deliverable: command-line generate.py that loads the model and produces text one sequence at a time.

Week 2 — Async server + naive batching

  • FastAPI server, /v1/chat/completions streaming.

  • Naive static batching: collect N requests, pad, run to completion together.

  • Yes this is bad — you’re building the baseline to measure improvement against.

  • Wire up async streaming SSE: engine puts tokens on a queue per request, server drains it.

  • Deliverable: curl-able server. Benchmark it — note the TTFT/throughput. Static batching gives you a reference “bad” number.

Week 3 — Continuous batching (the Orca step)

  • Scheduler loop: every iteration, remove finished sequences, admit waiting ones.

  • Handle prefill vs decode in the same batch:

    • MVP option A: run prefill in a dedicated step (no mixing), keeps kernels simple, hurts ITL.

    • MVP option B: mixed batches — needs your attention kernel to handle varying seq lens. Do B if you can.

  • Selective batching (Orca-style): only linear layers batch; attention runs per-sequence.

  • Deliverable: throughput jump of 3–10× over Week 2. Document it.

Week 4 — Paged KV cache

  • Allocate a big HBM buffer at startup: num_blocks × block_size × 2 × num_layers × num_kv_heads × head_dim × dtype_size.

  • Pick block_size = 16 (vLLM default).

  • Free list of physical block IDs.

  • Per-sequence block table: list[int] (logical block i → physical block ID).

  • When a sequence needs another block (its last block filled), allocate one from free list; on completion, return all its blocks.

  • Rewrite attention to gather from block table (start with the simplest possible Triton kernel, or use FlashInfer’s paged attention as a shortcut for MVP).

  • Deliverable: engine survives sequences with wildly different lengths without fragmentation; max concurrent sequences jumps.

Week 5 — Prefix caching

  • Hash each block’s token prefix (rolling hash: hash(prev_block_hash, tuple(block_tokens))).

  • Global dict[hash → block_id].

  • On prefill: walk the prompt in block_size chunks, look up hash, if hit → point block table at existing block, skip the compute for those tokens.

  • Reference-count blocks; free only when refcount hits 0.

  • Deliverable: replay a chat trace (prompt + follow-up) — the second turn’s TTFT should collapse. Log cache hit rate.

Week 6 — Benchmark harness + one polish item

  • Write bench_serve.py: Poisson arrivals at configurable rate, ShareGPT prompt distribution (grab the ShareGPT JSON), output p50/p95/p99 for TTFT and ITL.

  • Match vllm bench serve output format so comparison is trivial.

  • Pick ONE polish: chunked prefill, CUDA graphs, or n-gram spec decoding. Ship it.

  • Deliverable: benchmark script that runs against both your engine and vLLM with a config flag.

Week 7 — vLLM comparison + honest write-up

  • Same hardware, same model, same request trace, same seed.

  • Full sweep: request rate 1, 2, 4, 8, 16, 32 req/s. Report the goodput curve.

  • Honest analysis: where are you 2× slower? Where are you close?

  • Common answers: your attention kernel is slower than FlashInfer (fine), your Python loop has more overhead (fine), your prefix hash is O(n) not O(1) (fixable).

  • Deliverable: blog post / README with tables + graphs + prose explaining every gap.

Week 8 — Ship + submit

  • Clean up repo. README with quickstart, architecture diagram, benchmark tables.

  • Publish blog post. Post on r/LocalLLaMA and Twitter/Hacker News.

  • Open a GitHub issue on vLLM asking a specific question you hit during the build — this seeds the OSS-contribution habit.

  • Deliverable: public artifact. This is the portfolio piece.

Reference minimal engines to study (in this order)

  1. GeeeekExplorer/nano-vllm — https://github.com/GeeeekExplorer/nano-vllm

    • ~1000 lines of Python. Mirrors vLLM’s API. Includes prefix caching, TP, CUDA graphs.

    • Their own benchmark on RTX 4070: vLLM 1361 tok/s vs nano-vllm ~1400 tok/s. Not directly comparable (nano is simpler) but the point stands — you can get real performance in ~1K lines.

    • Read this before you start. Steal the file layout. Do not steal the implementation — you’ll learn nothing.

  2. changjonathanc/flex-nano-vllm — https://github.com/changjonathanc/flex-nano-vllm

    • FlexAttention (PyTorch’s block-sparse attention primitive) based. Great if you want to skip writing a paged attention kernel from scratch and understand FlexAttention instead. Blog post: “vLLM flex attention from scratch”.

  3. karpathy/nanoGPT — you already know it. Reference for clean PyTorch model code.

  4. vLLM V1 source itself — read the specific files listed in 10_vllm_source_map.md when you hit a design question. Don’t read it linearly; use it as a reference.

  5. SGLang source — read specifically python/sglang/srt/mem_cache/radix_cache.py when you get to Week 5. It’s the cleanest implementation of prefix caching you’ll find.

Design decisions you’ll face (and the “right” answer for MVP)

Question

Simple / Right for MVP

Later

Async engine loop in own process?

No, thread + asyncio.Queue

Yes (ZMQ, matches vLLM V1)

Custom paged attention kernel?

No, use FlashInfer

Yes, write Triton kernel

Block size?

16

Tunable (measure it)

Prefix cache data structure?

dict[hash] → block

Radix tree (matches SGLang)

Prefill/decode mixing?

Separate steps first, mix in Week 6

Chunked prefill with token budget

Preemption?

Recompute (cheap to implement)

Swap-to-CPU as second option

Sampling?

Greedy + top-p is enough

Full stack incl. min-p, logit bias

Structured output?

Skip for MVP

Wire XGrammar in Week 6 polish

Speculative decoding?

Skip for MVP

N-gram is easiest to add

TP / multi-GPU?

No — one GPU

Only after MVP ships

Quantization?

fp16/bf16 only

GGUF / AWQ integration later

Rule: every “later” is a follow-up blog post. Ship the MVP first.

Acceptance criteria (the exit gate)

  1. Engine sustains ≥50 concurrent sequences with continuous batching and paged KV at 8k context (Qwen 2.5 7B on a single 24GB GPU).

  2. Streaming SSE output is byte-identical to openai Python client parsing.

  3. On the ShareGPT trace at moderate load (~8 req/s), you’re within 3× of vLLM’s throughput. Being within 2× is excellent; within 1.5× is great.

  4. Prefix cache hit rate on a repeated-prefix trace matches vLLM’s within 5%.

  5. Public repo + benchmark write-up.

  6. You can walk someone through the request lifecycle from memory: arrive → tokenize → schedule (with block allocation, prefix hash check) → prefill → decode iterations → detokenize + stream → free blocks. No notes.

What this project buys you

  • study answer to every serving-systems question you’ll ever be asked. “How does continuous batching work?” — you built it.

  • Foundation for OSS contributions: you now understand vLLM’s scheduler well enough to find real bugs. The next portfolio step (a merged PR) becomes tractable.

  • Zoho work leverage: you can propose and implement a prefix-cache-aware routing layer for your agentic products with credibility — you know exactly how the underlying primitive works.

  • The habit that matters: predict the number, measure the number, explain the gap. Every scheduler decision, every kernel choice — hypothesis, arithmetic, benchmark, blog post.

Ship it in eight weeks. It is the fulcrum of everything after.