11 — SGLang Source Map

Reading SGLang alongside vLLM is a training exercise in comparative systems design. Same problem space, different aesthetic. SGLang started from RadixAttention and grew outward; vLLM started from PagedAttention and grew outward. The two centers of gravity show in the code.

Repo: https://github.com/sgl-project/sglang · Home: LMSYS (UC Berkeley alumni project) · Powers 400K+ GPUs at xAI, NVIDIA, AMD, LinkedIn as of 2026.

Why read SGLang if you already read vLLM

  1. RadixAttention is cleaner in SGLang than the equivalent in vLLM. If prefix caching matters to you (and given your Zoho agentic workloads, it does), this is the reference implementation to study.

  2. The scheduler is simpler. SGLang made different tradeoffs than vLLM V1 — fewer processes, less IPC, more direct. Reading it will make you appreciate what V1’s multi-process design bought and cost.

  3. Structured output is deeply integrated. SGLang has a first-class program-level structured decoding API, not just a per-request response_format. If you care about the frontend of structured output (not just the FSM masking), read SGLang.

  4. You’ll find bugs. Two codebases attacking the same problem let you triangulate. If vLLM does X and SGLang does Y, one of them is wrong (or the tradeoff is deliberate) — either way, you learn.

Repository layout at a glance

sglang/
├── python/sglang/
│   ├── srt/                     ← THE runtime (this is what you read)
│   │   ├── managers/            ← scheduler + tokenizer + detokenizer processes
│   │   ├── mem_cache/           ← RadixAttention lives here
│   │   ├── model_executor/      ← forward pass, CUDA graphs
│   │   ├── layers/              ← attention backends, MoE, quant
│   │   ├── constrained/         ← XGrammar / Outlines integration
│   │   ├── speculative/         ← EAGLE, MTP, n-gram
│   │   ├── entrypoints/         ← HTTP servers (OpenAI + native)
│   │   └── server_args.py       ← configuration surface
│   └── lang/                    ← the SGLang frontend DSL (fork points, gen, select)
└── docs/

Rule of thumb: srt/ = the serving runtime (what corresponds to vLLM’s engine). lang/ = the DSL layer above it. For inference-engineer purposes, read srt/ first and treat lang/ as awareness-level.

The reading plan (5 sessions × ~1 hour)

Session 1 — The scheduler (the entry point)

File

Read for

Key symbols

python/sglang/srt/managers/scheduler.py

The main scheduler loop. Compare to vLLM’s v1/core/sched/scheduler.py.

Scheduler, event_loop_normal, event_loop_overlap, run_batch, get_next_batch_to_run

python/sglang/srt/managers/schedule_batch.py

The batch data structure.

ScheduleBatch, Req, prepare_for_prefill, prepare_for_decode

python/sglang/srt/managers/schedule_policy.py

Admission/priority policy.

SchedulePolicy, PrefillAdder

What to look for:

  • event_loop_overlap — SGLang’s key optimization: overlap CPU-side scheduler work with GPU-side model execution. Compare to vLLM V1’s approach (multi-process ZMQ). Two different solutions to the same problem.

  • How PrefillAdder decides which waiting requests to admit. Its budget accounting is where chunked prefill lives in SGLang.

Session 2 — RadixAttention (the reason to read SGLang)

This is the highlight of the tour. Take your time here.

File

Read for

Key symbols

python/sglang/srt/mem_cache/radix_cache.py

The radix tree.

RadixCache, TreeNode, match_prefix, insert, evict, inc_lock_ref, dec_lock_ref

python/sglang/srt/mem_cache/memory_pool.py

The physical KV pool.

ReqToTokenPool, TokenToKVPool

python/sglang/srt/mem_cache/base_prefix_cache.py

The abstract cache interface (there’s also ChunkCache and HiRadixCache).

BasePrefixCache

What to look for:

  1. TreeNode structure — note that keys are token-id sequences (bytes-like), children are a dict, and each node holds the physical KV block IDs for the tokens it covers. Compare mentally to a trie for strings.

  2. match_prefix(tokens) — walks the tree greedily, splitting nodes when a partial match is found. Returns:

    • the longest matched prefix (token IDs already cached)

    • the KV block IDs to reuse

    • the leaf node reached

  3. insert(tokens, kv_indices) — called after prefill completes. Extends the tree with new tokens.

  4. Lock reference counting (inc_lock_ref/dec_lock_ref) — nodes referenced by RUNNING requests can’t be evicted. This is the concurrency-safe eviction discipline.

  5. evict(num_tokens) — LRU eviction over leaves only (interior nodes’ KV is still needed by descendants). This is the elegant part.

Exercise: draw the tree state after 3 requests share a system prompt and then diverge. Note the shared-prefix nodes’ lock counts. Now delete one request. What can be evicted?

Read the LMSYS blog alongside: https://www.lmsys.org/blog/2024-01-17-sglang. The diagrams there map directly to this code.

Session 3 — Model executor & CUDA graphs

File

Read for

Key symbols

python/sglang/srt/model_executor/model_runner.py

The forward-pass driver.

ModelRunner, forward, forward_extend (prefill), forward_decode

python/sglang/srt/model_executor/cuda_graph_runner.py

CUDA graph capture and replay.

CudaGraphRunner, capture, replay

python/sglang/srt/model_executor/forward_batch_info.py

Per-batch metadata (block tables, positions).

ForwardBatch, ForwardMode

What to look for:

  • SGLang splits forward into forward_extend (prefill or chunked prefill) and forward_decode. vLLM V1 unifies these more aggressively. Both approaches work; note the code-clarity vs performance tradeoff.

  • CUDA graph capture happens for decode-shaped batches only, at bucketed batch sizes. Same pattern as vLLM but implemented independently.

Session 4 — Attention backends & speculative decoding

File

Read for

Key symbols

python/sglang/srt/layers/attention/flashinfer_backend.py

FlashInfer-based attention.

FlashInferAttnBackend

python/sglang/srt/layers/attention/triton_backend.py

Triton attention fallback.

TritonAttnBackend

python/sglang/srt/speculative/eagle_worker.py

EAGLE draft + verify.

EAGLEWorker, draft, verify

python/sglang/srt/speculative/spec_info.py

Speculative state.

SpecInfo

What to look for:

  • How the attention backend consumes ForwardBatch (block tables + positions) — same paged-KV gather pattern as vLLM.

  • EAGLE’s tree-attention verify step — the draft tree of candidate tokens is verified with a single forward pass. Note how the tree is flattened into a batch.

Session 5 — Structured output & entrypoints

File

Read for

Key symbols

python/sglang/srt/constrained/xgrammar_backend.py

XGrammar integration.

XGrammarGrammar, fill_next_token_bitmask

python/sglang/srt/constrained/outlines_backend.py

Outlines integration.

OutlinesGrammar

python/sglang/srt/entrypoints/http_server.py

Native SGLang HTTP server.

launch_server, endpoint handlers

python/sglang/srt/entrypoints/openai/serving_chat.py

OpenAI-compatible surface.

OpenAIServingChat

What to look for:

  • The XGrammar mask application: it’s a bitmask.masked_fill(logits, -inf) at the last moment. Same primitive as vLLM — different plumbing.

  • The OpenAI-compatible layer is thinner than in vLLM. SGLang’s native API assumes you’ll use their frontend DSL for programmatic use.

SGLang vs vLLM — architectural diff at a glance

Aspect

vLLM V1

SGLang

Process model

Multi-process (API + EngineCore + workers), ZMQ IPC

Fewer processes; overlap-based scheduling

Prefix cache

Hash-chained blocks in dict

Radix tree of token sequences

Attention default

FA3 / FlashInfer

FlashInfer / Triton

Spec decoding

Mature (EAGLE-3, ngram, medusa)

EAGLE, MTP, ngram — experimental for some

Structured output

XGrammar (default), Outlines, llguidance

XGrammar (default), Outlines

Frontend DSL

None (OpenAI-only)

SGLang lang/ DSL (fork, gen, select)

Multi-modal

Yes

Yes (increasingly — llava, qwen-vl families)

Best at

Broadest hw support, biggest ecosystem, spec decode

Prefix-heavy workloads, agentic, program-level structured output

Rule for choosing (recap of 05_radixattention_sglang.md):

  • Agentic / long-system-prompt / multi-turn tool-calling workloads → SGLang first, benchmark vs vLLM.

  • Everything else / mixed workloads / need latest spec decoding → vLLM first.

Key contributors

  • Lianmin Zheng (@merrymercy) — lead author of SGLang and RadixAttention paper.

  • Ying Sheng (@Ying1123) — co-lead, structured output.

  • Zhiqiang Xie, Yineng Zhang — core runtime.

  • Ke Bao, Yichuan Wang — kernels, quantization.

Community home: LMSYS Slack/Discord. Watch the sglang-project org, not individual repos.

Good-first-issue clusters in SGLang

  1. New model support — same as vLLM: copy the closest existing model file in srt/models/, adapt.

  2. Frontend DSL primitives — add a new sglang.gen() variant, a new sampling method. Isolated in lang/.

  3. RadixCache profiling — add metrics: hit rate, tree depth, eviction rate. High-value contribution since observability is often thin.

  4. XGrammar bug reports — you’ll find edge cases just by using it. File them.

  5. Documentation — SGLang’s docs lag its code more than vLLM’s. Well-written subsystem docs are welcomed.

Reading discipline

  • Read radix_cache.py line by line. It’s the most instructive single file in modern inference engines. Don’t skim.

  • Build a mental diff with vLLM. For every SGLang design decision, ask: what would vLLM do here, and why the difference?

  • Use both engines locally. Run the same trace through both, compare TTFT/ITL/prefix-cache-hit-rate. The numbers explain the code better than the comments.

SGLang and vLLM will keep converging in features and diverging in aesthetic. Reading both makes you the person who can tell a team which one to bet on for their specific workload — with evidence.