Open Source Inference Engines: vLLM / SGLang / FlashInfer / llama.cpp

This is where the field actually happens. A merged PR here is the credential.

---

The four repos and their positioning

Engine	Primary audience	Language	Where it dominates	GitHub
vLLM	Production serving, cloud	Python + CUDA/Triton + C++	The de facto reference engine; most citations	https://github.com/vllm-project/vllm
SGLang	Structured / agentic workloads, high-throughput serving	Python + CUDA/Triton	RadixAttention, structured decoding, DeepSeek serving	https://github.com/sgl-project/sglang
FlashInfer	Attention kernel library (used by both above)	C++/CUDA + Python	Paged/ragged attention, JIT compilation	https://github.com/flashinfer-ai/flashinfer
llama.cpp	Local / consumer / on-device	C++ + Metal + CUDA	GGUF ecosystem, CPU inference, Apple silicon, edge	https://github.com/ggml-org/llama.cpp

A note on the llama.cpp URL: the project moved from ggerganov/llama.cpp to ggml-org/llama.cpp in 2025. The old URL redirects.

---

The universal PR pipeline (works for all four)

1. Use it in anger for 4+ weeks. No shortcut. You cannot contribute to what you haven’t stress-tested.

2. Read the CONTRIBUTING.md and dev docs. Every project. Every time.

3. Skim closed PRs. Filter by label “good first issue”, “help wanted”, “performance”. You learn the review culture and the maintainers’ style before opening one.

4. Reproduce a recent bug. Pick an open issue with bug label, no assignee, no recent activity. Reproduce it locally. Post: “Reproduced on commit X, environment Y, minimal repro attached.” This alone is worth reputation.

5. Fix that bug (if within scope). Small PR. Follow their conventions to the letter: commit message format, changelog entry, test naming.

6. Grow. Docs → benchmark scripts → small kernel opts → feature-area ownership.

The single strongest first PR: a doc-clarification PR that fixes a genuine confusion you experienced during setup, backed by the exact command that failed for you. Maintainers universally love these. They also let you learn the PR mechanics on a low-risk change.

---

1. vLLM

URL: https://github.com/vllm-project/vllm

The lay of the land (2026)

vLLM V1 is the current architecture (V0 was deprecated late 2024/2025). Async engine core, separate scheduler/executor processes, ZMQ IPC. It ships PagedAttention, chunked prefill, prefix caching, speculative decoding (n-gram, EAGLE-1/2/3, Medusa), disaggregated P/D via KV connectors (LMCache, NIXL), structured decoding via XGrammar/Outlines/llguidance, TP/PP/EP/DP scaling.

Good-first-issue areas (rank-ordered)

1. Docs. vLLM’s docs move fast and lag behind code. Fixing a broken example or an out-of-date CLI flag is a real contribution.

2. Benchmark scripts. benchmarks/ is under-loved. Adding a workload profile or fixing a stale reference config is welcome.

3. Model support. Adding a new HF model (or a variant) is a well-scoped first PR — there’s a template and reference PRs to copy.

4. Quantization method integration. Bringing a new quant format into vllm.model_executor.layers.quantization is bounded and impactful.

5. Tests for edge cases. Every bug fix should have a regression test; missing tests are visible in coverage.

6. Attention backends. Later stage: writing/optimizing a FlashInfer or Triton attention kernel path.

7. Scheduler heuristics. Advanced: chunked-prefill token budget tuning, preemption policies.

Where NOT to start

• Core scheduler refactors (owned by a small core team; long review cycle).

• Speculative decoding internals (fast-moving, senior contributor territory).

• V1 engine core rewrite proposals (nope).

Community touchpoints

• Slack: vLLM maintains a public Slack; invite link is in the README (rotating URL, always check current).

• Office hours: vLLM runs bi-weekly public dev meetings. Announced in Slack #general. Recordings on YouTube channel https://www.youtube.com/@vllm-project.

• Blog: https://blog.vllm.ai

• Discord: vLLM has a Discord too (linked from README). Slack tends to be more active for dev discussion.

Top contributors to follow (verifiable via git shortlog -sn on the repo)

• Woosuk Kwon (@WoosukKwon) — co-creator, PagedAttention lead author

• Zhuohan Li (@zhuohan123) — co-creator

• Simon Mo (@simon-mo) — core, project lead

• Cade Daniel — speculative decoding

• Roger Wang (@ywang96) — multimodal

• Nick Hill (@njhill) — engine core

• Michael Goin (@mgoin) — quantization, Neural Magic

• Robert Shaw (@robertgshaw2-redhat) — quantization, serving

• Kaichao You (@youkaichao) — torch.compile integration

---

2. SGLang

URL: https://github.com/sgl-project/sglang

The lay of the land

SGLang originated from LMSYS (Chatbot Arena team, UCB) and grew into a serious production engine competitive with vLLM, especially strong at RadixAttention prefix caching, structured decoding, and DeepSeek-V3/R1-scale MoE serving with expert parallelism.

Good-first-issue areas

1. Docs + tutorials — same as vLLM, docs lag rapid dev.

2. Benchmark reproduction — SGLang’s bench_*.py is fertile.

3. New model support — well-scoped.

4. RadixAttention tests + eviction edge cases.

5. Structured decoding integrations (XGrammar, llguidance interop).

6. DeepSeek-serving optimizations (huge current focus).

7. Router / KV-aware routing improvements.

Community touchpoints

• Slack: public invite from README.

• Bi-weekly meetings: announced in Slack + issues.

• Blog: https://lmsys.org/blog/ (SGLang posts) and dedicated SGLang blog when new features land.

Top contributors to follow

• Lianmin Zheng (@merrymercy) — co-creator, LMSYS/Chatbot Arena

• Ying Sheng (@Ying1123) — co-creator, RadixAttention

• Yineng Zhang (@zhyncs) — core, perf

• Byron Hsu (@ByronHsu) — Liger Kernel, integrations

• Ke Bao (@ispobock) — core

---

3. FlashInfer

URL: https://github.com/flashinfer-ai/flashinfer

The lay of the land

FlashInfer is the attention kernel library used by vLLM and SGLang for paged/ragged attention on Ampere/Hopper/Blackwell. Also does sampling, top-p/top-k, and increasingly non-attention kernels. MLSys 2025 best paper (arXiv 2501.01005). Under active development.

Good-first-issue areas

1. Tests on new hardware / new shapes. FlashInfer supports many attention variants (MHA/GQA/MLA, paged/ragged, sliding window, causal, etc.) — test matrix is enormous.

2. Benchmarks on new GPU generations (Ada, Blackwell).

3. Python API polish — the C++/Python boundary always has friction.

4. New sampling algorithms (min-p, DRY, XTC).

5. Docs and examples.

6. Advanced: JIT template additions for new attention variants.

Community touchpoints

• GitHub Discussions: primary venue.

• Twitter: author Zihao Ye posts development updates.

• Blog: https://flashinfer.ai/ has technical posts (Cascade Inference, etc.).

• GPU MODE Lec 40 — authored talk.

Top contributors to follow

• Zihao Ye (@yzh119) — CMU PhD, primary author

• Lequn Chen (@abcdabcd987) — co-author

• Vinay Joshi (@vinx13) — core

• Tianqi Chen — advisor (CMU/OctoAI/CatalystML)

---

4. llama.cpp

URL: https://github.com/ggml-org/llama.cpp

The lay of the land

Georgi Gerganov’s project, moved to the ggml-org GitHub org in 2025. The reference implementation for local inference: GGUF format, k-quants/i-quants, CPU (AVX/AMX/ARM/Metal) + GPU (CUDA/HIP/SYCL/Vulkan/Metal/MUSA) backends. llama-server is a production HTTP endpoint. Ollama and LM Studio wrap it. Codebase is C++ with minimal external deps — approachable if you can read modern C++.

Good-first-issue areas

1. Docs for CLI tools (llama-cli, llama-server, llama-bench, llama-quantize, llama-perplexity).

2. New model architecture support (convert_hf_to_gguf.py + src/llama-arch.cpp + src/llama-model.cpp). Well-scoped first task.

3. CPU kernel optimizations for a specific quant format on a specific ISA (AVX-512, AVX2, NEON, SVE, AMX).

4. GPU backend kernels (CUDA/Metal/Vulkan/SYCL) for specific ops.

5. llama-server features: OpenAI API compatibility gaps, streaming edge cases.

6. Sampling improvements — new sampler additions have been welcome historically.

7. Benchmark automation.

Community touchpoints

• Issues + Discussions: primary venue.

• The #llama.cpp channel on the ggml.ai Discord (linked from README when open).

• HuggingFace GGUF community — quant packagers coordinate via HF + Reddit.

Top contributors to follow

• Georgi Gerganov (@ggerganov) — creator; less day-to-day now but still reviews.

• slaren — GPU / CUDA backend.

• ikawrakow — quantization (k-quants, i-quants); now maintains ik_llama.cpp fork with newer quant work.

• compilade — architecture + Mamba/hybrid models.

• ngxson — llama-server, HF integration.

• CISC, JohannesGaessler, 0cc4m — kernel + backend work.

---

Bonus repos you should also watch (lurk-only or targeted)

• TensorRT-LLM — https://github.com/NVIDIA/TensorRT-LLM — different contribution culture (NVIDIA-owned). Awareness for prod comparison.

• NVIDIA Dynamo — https://github.com/ai-dynamo/dynamo — KV-aware routing / disaggregation layer above engines.

• llm-d — https://github.com/llm-d/llm-d — K8s-native distributed serving.

• LMCache — https://github.com/LMCache/LMCache — KV cache offload / sharing.

• NIXL — https://github.com/ai-dynamo/nixl — KV transfer library.

• llm-compressor — https://github.com/vllm-project/llm-compressor — quantization toolkit vLLM uses.

• AutoGPTQ / AutoAWQ — https://github.com/AutoGPTQ/AutoGPTQ , https://github.com/casper-hansen/AutoAWQ.

• exllamav2 / exllamav3 — https://github.com/turboderp-org/exllamav2 — single-GPU enthusiast serving.

• TorchTitan — https://github.com/pytorch/torchtitan — reference distributed training with FSDP2/TP/PP.

• nanotron — https://github.com/huggingface/nanotron — HF’s 3D-parallel trainer, digestible codebase.

• verl — https://github.com/volcengine/verl — RLHF/GRPO with vLLM inside training loop.

• Modular MAX / Mojo — https://github.com/modular/max — awareness of the alt-stack.

---

First PR timeline (aligned to 13-month plan)

Month	Action
M4	Install all 4 engines, run their benchmarks on your GPU. Read README + CONTRIBUTING for each.
M6	Docs PR to one of them (fix a real thing you tripped on).
M8	Reproduce + report a filed bug in one project.
M10	Small perf/behavior fix PR — the first “real” merge.
M13	Feature-area contribution (new model support, new quant integration, new attention shape).
M18+	Sustained ownership of a feature area — the job-offer tier (see ../12_portfolio/08_rung_8_sustained_contribution.md).

---

PR hygiene rules (survival guide)

• Small. One thing per PR. Big PRs die.

• Reproducible. Include benchmark command + before/after numbers if perf-related.

• Tested. New behavior needs a test; bug fix needs a regression test.

• Formatted. Run the project’s pre-commit / ruff / clang-format. Don’t make reviewers care.

• Documented. Update the doc string, the changelog entry, the CLI help.

• Responsive. Reply to review comments within 48 hours or your PR rots.

• Humble in the description. “Adds X. Fixes #Y. Benchmarks: [table]. Would appreciate review from @Z who owns this area.”

• Never argue with a reviewer in public. If you disagree, ask for clarification, propose alternatives, or take it offline. Maintainer patience is finite.