Phase 4 — Inference Engine Internals¶

Months 8–12. This is the heart of the roadmap. Everything before this (systems, transformers, GPU kernels, attention) was substrate. Everything after (quantization, distributed serving, production) is elaboration on the mechanisms you learn here.

Why this phase is the whole roadmap in miniature¶

The job title “Inference Engineer” fundamentally names the person who owns the box between the HTTP request and the tensor core. That box has four parts:

A scheduler — decides which requests run this iteration.
A memory manager — allocates KV cache blocks, tracks who owns what.
A model runner — actually executes the forward pass.
A metrics surface — because you can’t optimize what you can’t measure.

vLLM and SGLang are just two very good implementations of those four parts. By the end of this phase you will have written all four yourself, and read enough vLLM V1 source that the codebase feels like a house you grew up in.

The reason this is the heart of the roadmap: every other phase now compiles down to a concrete “why”. You learned FlashAttention (Phase 3) — here you find out why the paged-KV variant exists. You learned KV cache arithmetic (Phase 1) — here you find out why fragmentation was the enemy. You learned continuous batching in the abstract — here you write the loop that does it.

The strategic wedge you personally hold¶

Read this carefully because it shapes what you emphasize.

You spend your day writing agentic harnesses and long-running automation services at Zoho. Multi-turn tool calls. Long system prompts. Retry loops. Traces that re-send history. In inference-serving terms this is the workload most punished by naive engines and most rewarded by prefix caching / RadixAttention (see 05_radixattention_sglang.md).

RadixAttention is your paper. More than any other item in this phase, it is the one you can walk into work on Monday, measure, and turn into a real win with a real number attached (“cache hit rate on our agent traffic is 74%; switching to prefix-cache-aware routing dropped p95 TTFT by 38%”). That number funds your credibility, which funds your license to do everything else in this roadmap on company time.

Treat file 05 as the single highest-ROI document in the phase.

The five ideas in this phase, ranked by permanence¶

Continuous batching (Orca) — iteration-level scheduling. Once you understand this, static batching feels absurd. This idea is not going anywhere for the next decade.
PagedAttention (vLLM) — treating KV cache like virtual memory. Also permanent; every serious engine has adopted paging.
Prefix caching / RadixAttention (SGLang) — sharing KV across requests with matching prefixes via a radix tree. Permanent; agentic workloads made this table stakes.
Chunked prefill (Sarathi-Serve) — merging prefill chunks into decode batches to eliminate ITL stalls. Permanent, though the token-budget knob is workload-tunable.
Speculative decoding (Leviathan/Chen → Medusa → EAGLE → MTP) — the only one of the five where the implementation is still shifting fast. Understand the mathematical framework (rejection sampling), then track EAGLE-3/MTP for current SOTA.

What “source-level fluency” means¶

By the end of this phase you should be able to:

Draw the vLLM V1 process diagram from memory: API server ↔ EngineCore (scheduler + block manager + model executor) ↔ worker processes.
Name the file paths for the scheduler (vllm/v1/core/sched/scheduler.py), the KV cache manager (vllm/v1/core/kv_cache_manager.py), the engine core loop (vllm/v1/engine/core.py), and the GPU model runner (vllm/v1/worker/gpu_model_runner.py).
Walk a request through end-to-end: arrival → tokenize → schedule → block allocation → prefill (possibly chunked) → decode iterations → preemption/eviction → stream → free.
Do the same for SGLang: python/sglang/srt/managers/scheduler.py, python/sglang/srt/mem_cache/radix_cache.py, srt/model_executor/.
Predict, on a whiteboard, how a config change (max-num-seqs, max-num-batched-tokens, enable-prefix-caching, enable-chunked-prefill, speculative-config) moves the latency-throughput curve, before running it.

Fluency is not memorizing lines. It’s knowing where to look and what question the code was written to answer.

The current-state snapshot (verified July 2026)¶

Because vLLM/SGLang move fast, snap a picture in your mind at start:

Project	State as of mid-2026	Where it wins
vLLM V1	Default engine since ~mid-2025; multi-process architecture; 66K+ GH stars; TPU backend GA; disaggregated P/D landing; production-grade EAGLE-2/3 speculative decoding.	Broadest hardware support (NVIDIA + AMD ROCm + TPU + Intel). Best OSS community. Best “just works” story.
SGLang	LMSYS-led; powers 400K+ GPUs at xAI/NVIDIA/AMD/LinkedIn; RadixAttention native; XGrammar native.	Prefix-heavy / agentic workloads; structured-output workloads; JSON tool-calling. Up to 6.4× vs vLLM at extreme prefix overlap; typically 20-40% lower TTFT on realistic multi-turn traffic.
TensorRT-LLM	NVIDIA-only, engine-build model. Best raw perf on H100/H200/B200 FP8 when workload is stable.	Highest tok/s per NVIDIA GPU; production behind Triton Inference Server; sub-100ms TTFT at 64-concurrent on H100.
HF TGI	Moved to maintenance mode December 2025.	Legacy deployments only. Do not choose for new work.
llama.cpp	ggml-org/llama.cpp; the entire local ecosystem depends on it; still the reference CPU + heterogeneous engine.	Local, CPU, Apple Silicon, mixed offload.

Flag (breaking change since seed doc): V1 is now default in vLLM. VLLM_USE_V1=1 is no longer needed. If you see docs referring to V0 code paths (vllm/engine/llm_engine.py in the top-level engine dir), those are legacy — the code you should read is under vllm/v1/.

Files in this directory (read in order)¶

01_metrics_language.md — the vocabulary of serving: TTFT, TPOT/ITL, goodput, the latency-throughput curve, how to actually run vllm bench serve and interpret it.
02_orca_continuous_batching.md — iteration-level scheduling with a worked example. This unlocks everything.
03_pagedattention_vllm.md — the block table walkthrough; why fragmentation was the enemy; file-by-file map of vLLM V1’s scheduler and block manager.
04_chunked_prefill.md — Sarathi-Serve: stall-free scheduling and the token-budget knob.
05_radixattention_sglang.md — your paper. Radix tree, LRU eviction, cache-aware routing, and the Zoho playbook to measure hit rate on your real traffic.
06_speculative_decoding.md — the full lineage from Leviathan/Chen through Medusa, EAGLE-1/2/3, MTP, and n-gram / prompt-lookup drafts.
07_structured_decoding.md — Outlines/XGrammar/llguidance mechanics: FSM compilation, per-step masking, and the Adaptive Token Mask Cache trick that made it nearly free.
08_cuda_graphs.md — why decode captures graphs, when eager is faster, how vLLM’s graph capture works in V1.
09_mini_engine_capstone.md — THE big project. Week-by-week build plan for the mini inference engine, with reference minimal repos to study.
10_vllm_source_map.md — file-by-file guide to reading vLLM V1 source. “Code is truth.”
11_sglang_source_map.md — same for SGLang.
12_llama_cpp_world.md — GGUF Q4_K_M bit layout, k-quants vs i-quants, imatrix, -ngl mechanics, CPU bandwidth math.
13_local_ecosystem.md — Ollama, LM Studio, llama-server, MLX, exllamav2/v3, r/LocalLLaMA cultural literacy.
14_projects.md — Phase 4 project ladder with acceptance criteria.

Exit criteria (do not advance until all three)¶

Mini-engine capstone shipped. Continuous batching + paged KV streams correct output under concurrent load. Latency-throughput curve plotted vs vLLM on identical hardware, with an honest accounting of the gap.
vLLM V1 lifecycle from memory. You can whiteboard the full path from request arrival to response completion, naming files and functions.
GGUF Q4_K_M at the bit level. You can draw the super-block layout (256 weights, 8 sub-blocks × 32 weights, 6-bit scales/mins) and explain why Q4_K_M mixes Q4_K + Q6_K per tensor.

Additionally: at least one benchmarking artifact in your public portfolio (a bake-off post, a mini-engine repo, or a merged docs/perf PR to vLLM/SGLang).

The two anchor blog posts (verified URLs)¶

Aleksa Gordić — “Inside vLLM: Anatomy of a High-Throughput LLM Inference System” — https://www.aleksagordic.com/blog/vllm (also mirrored at https://vllm.ai/blog/2025-09-05-anatomy-of-vllm). Reviewed by Nick Hill (RedHat, vLLM core contributor). Covers LLM engine + engine core, chunked prefill, prefix caching, guided and speculative decoding, disaggregated P/D. Read this three times over the phase.
kipply — “Transformer Inference Arithmetic” — https://kipp.ly/blog/transformer-inference-arithmetic (2022). Every napkin-math exercise in this phase traces back to this post. Read it once at the start; return to it whenever a benchmark surprises you.

Two secondary must-reads:

vLLM V1 alpha blog (Jan 27, 2025) — https://vllm.ai/blog/2025-01-27-v1-alpha-release
LMSYS SGLang blog (Jan 17, 2024) — https://www.lmsys.org/blog/2024-01-17-sglang

The mindset shift this phase demands¶

Phase 3 (attention kernels) rewards depth-first focus on one algorithm. Phase 4 rewards the opposite: breadth of mechanisms, held in tension. You will need to reason about scheduling and memory management and sampling and SLOs simultaneously, because every real production tuning decision involves at least three of them.

The best way to develop that muscle: run benchmarks constantly, and predict the number before you look. This is not optional. If you never predict, you never build the model.

Move on when you can look at a vllm bench serve result — TTFT p50, TTFT p99, TPOT p50, throughput at concurrency N — and immediately diagnose the operating regime (compute-bound? memory-bound? queueing? KV-cache pressure? preemption?) before reading the config.