14 — Phase 4 Projects (Acceptance Criteria)¶

Every project here is public. Every project ships with a benchmark script, a data table, and a written explanation of the numbers. That triple — code, data, prose — is the deliverable. Not “I tried X.” “Here is X, here are its measured numbers on Y hardware, here is why they came out that way.”

The five Phase 4 projects, ranked by portfolio signal¶

P4.1 — Mini inference engine (capstone)
P4.2 — vLLM V1 source read-through with written architecture doc
P4.3 — Quant bake-off (7–8B model × 5+ formats)
P4.4 — Prefix-cache measurement at Zoho (work-integrated; internal write-up)
P4.5 — First OSS PR to vLLM / SGLang / llama.cpp / FlashInfer

Roughly parallel. P4.1 is the anchor everything else supports.

P4.1 — Mini Inference Engine (capstone)¶

Full spec: see 09_mini_engine_capstone.md. Restated acceptance here.

Deliverables:

GitHub repo, permissive license.
README: quickstart, architecture diagram, benchmark tables, honest limitations.
Blog post (personal site or Medium): tell the story, show the numbers, explain the gaps.
Reddit/HN post inviting critique.

Acceptance criteria (all must be met):

Continuous batching — sustains ≥50 concurrent sequences at 8k context on a single 24GB GPU with a 7B model.
Paged KV cache — evictable, refcount-safe, no leaks after 1000-request stress test.
Prefix caching — hit rate on a repeated-prefix trace is within 5% of vLLM’s on the same trace.
Streaming SSE output — byte-compatible with the openai Python client.
Benchmark harness — reproduces vllm bench serve output format for TTFT/ITL p50/p95/p99.
vLLM comparison — within 3× vLLM throughput at 8 req/s ShareGPT load. Within 2× is excellent.
Documented lifecycle — you can walk anyone through arrival → tokenize → schedule → prefill → decode → detokenize → free, unassisted.

Time budget: 8 weeks, ~10 hrs/week.

Success signal: you get specific technical feedback on Reddit/HN. Not just kudos — people noticing that your block manager’s LRU implementation would fragment under X workload. That’s the sign the project is real.

P4.2 — vLLM V1 Source Read-Through + Architecture Write-Up¶

Read 10_vllm_source_map.md’s 7 sessions. Then write your own architecture document. Not a summary of the source — an explainer for a developer who’s never seen it.

Deliverables:

A 3000–5000-word blog post on your site titled something like “vLLM V1 for people who want to modify it.”
Diagrams: at least the request lifecycle, the scheduler state machine, and the KV cache manager’s data structures.
One “here’s what surprised me” section — non-obvious findings that only come from reading code.
Cite specific file paths and (commit-pinned) line numbers.

Acceptance criteria:

Diagrams cover: process topology, request state machine, KV block lifecycle, spec-decoding data flow.
Every claim references a file + commit hash.
At least three “surprises” — things not obvious from the blog posts or paper.
A section on “what I’d change” — opinionated critique with rationale. This is what makes it a portfolio piece and not a summary.

Time budget: 3–4 weeks, in parallel with P4.1 or after.

Success signal: a vLLM maintainer links to your post in a Slack channel or issue thread. It happens.

P4.3 — Quantization Bake-Off¶

Full spec: see 12_llama_cpp_world.md Part 5, and Phase 5 details in the roadmap.

Bring it up in Phase 4 with the local-inference variants (GGUF K-quants + I-quants). Expand in Phase 5 with datacenter quants (AWQ, GPTQ, FP8).

Deliverables:

One 7–8B model at 8–12 quant variants.
Metrics: perplexity, KL to fp16, 3–5 lm-eval-harness tasks, tok/s across hardware configs, file size.
Blog post + reproducible benchmark scripts.
One “vibe check” section with actual model outputs (readers love this).

Acceptance criteria:

All numbers reproducible from committed scripts — CI-runnable would be ideal.
Hardware disclosed in full (GPU, RAM, driver, kernel).
A “pick this format if…” decision table.
At least one surprise (there’s always one — e.g., a specific model where IQ4_XS beats Q4_K_M on MMLU by 2 points, or where Q3_K_M is better than expected).

Time budget: 2–3 weeks.

Success signal: post gets 100+ upvotes on r/LocalLLaMA; Bartowski or Unsloth links to your table.

P4.4 — Prefix-Cache Measurement at Zoho (work-integrated)¶

This is your Trojan horse. Highest ROI project on the entire list because it converts your day-job traffic into learning + internal credibility + a real production win.

Deliverables (internal, but shape a public sanitized version):

Instrumentation on your existing agentic-harness traffic to log token-prefix hit rates against a simulated radix cache.
Report: “our multi-turn agent traffic has X% prefix-shareable tokens; if we deployed SGLang/vLLM-V1 with prefix caching, we would see Y% throughput improvement and Z% TTFT reduction.”
A/B test proposal or actual rollout, if you can get it approved.
Public write-up (sanitized, aggregated numbers, no customer data): “Prefix caching in production agentic workloads: measured impact.”

Acceptance criteria:

Real traffic sample of ≥10k requests analyzed.
Hit-rate numbers segmented by request pattern (chat continuation vs tool-loop vs new session).
Explicit ROI estimate: “this saves $X/month in inference compute, or reduces p95 latency by Y ms.”
Concrete next step: proposal to switch engine, or to add a prefix-aware router in front of existing serving.

Time budget: 2–3 weeks part-time, aligned with your normal Zoho work.

Success signal (internal): your team adopts the proposal. This is the tier of contribution that gets you seen as an inference platform owner, not a consumer.

Success signal (external): the sanitized post — with real numbers from a real enterprise workload — is rare content. Sanitized properly, it’s a strong hiring signal for any team building agentic infrastructure.

P4.5 — First OSS Contribution¶

The end-state of Phase 4 is a merged PR to vLLM, SGLang, FlashInfer, or llama.cpp.

Path (in ascending order of ambition):

Level 0 — Great issue¶

File a well-researched, reproducible bug report or feature request. Not fixed yet, but the quality of the issue is itself a signal.

Acceptance: minimal repro, expected vs actual, hardware/software versions, hypothesis about root cause. Maintainer engagement (not just a bot response).

Level 1 — Docs PR¶

Add or clarify documentation on a subsystem you now understand from reading source. Zero-risk, high-signal. This is how many contributors get their first merge.

Acceptance: merged. Bonus: the doc referenced by a maintainer in a subsequent issue.

Level 2 — Test PR¶

Add a test that captures a subtle behavior you noticed while reading code. Regression tests for edge cases in the scheduler, KV manager, or samplers are always welcome.

Acceptance: merged, with a maintainer commenting that it caught something they hadn’t noticed.

Level 3 — Bug fix¶

Fix a real bug with a test. Look for issues labeled good first issue or help wanted. Or bring a bug you found in your mini-engine build back upstream.

Acceptance: merged, cited in release notes.

Level 4 — Feature¶

Add support for a new model, a new sampling method, a new metric export, a new logit processor. This is the tier where you start being known in the project.

Acceptance: merged, and you get pinged on future related issues.

Level 5 — Area ownership¶

Sustained contribution to one area (e.g., “chunked prefill edge cases”, “XGrammar integration”, “prefix cache observability”). Usually 6–12 months of Level-3/4 contributions in the same file cluster.

Acceptance: you get review authority on PRs touching your area. This is the job-offer tier from these projects.

Time budget for Level 1 (Phase 4 exit): 1 week once you’ve done the source read-through.

Time budget for Level 3 (Phase 5 exit): 2–3 weeks over Phase 4/5.

Time budget for Level 5: the roadmap. This is what you’re building toward.

Cross-project discipline¶

Applies to every project above:

Lab notebook per project. Every experiment: hypothesis → predicted number → measured number → explanation. This is the entire skill compressed to one habit.
Benchmark hygiene. Warmups, locked clocks (nvidia-smi -lgc <freq>), pinned engine versions, deterministic seeds where possible, p99s not means, and scripts committed to the repo. Numbers you can’t reproduce are gossip.
Retro at project end. One page: what worked, what didn’t, what surprised you, what to do differently next time. This is where compounding learning lives.
Publish or it didn’t happen. Blog post, repo, or Reddit thread — every project has a public artifact. If you can’t explain it publicly, you haven’t finished it.
One project in flight, one in review, one in planning. Don’t parallelize more than three; you lose depth.

Phase 4 exit — the composite portfolio state¶

When Phase 4 is done, your public portfolio has:

A working mini-inference-engine repo with benchmark write-up (P4.1).
A vLLM V1 architecture blog post with diagrams (P4.2).
A quant bake-off post on r/LocalLLaMA (P4.3).
A sanitized prefix-caching write-up from real Zoho traffic (P4.4).
At least one merged PR (docs or test level) in vLLM, SGLang, or llama.cpp (P4.5 Level 1–2).

You can now walk into any inference-engineer study and answer:

“How would you design a serving system for 70B at 10 req/s SLO?” — with a concrete plan citing your own benchmark data.
“What’s the difference between vLLM V1’s scheduler and SGLang’s?” — with the file references.
“When does speculative decoding help?” — with acceptance-rate arithmetic and a workload-dependent decision.
“How does prefix caching change with agentic workloads?” — with production numbers.

That portfolio + that answer set is not entry-level. It’s the mid-to-senior inference engineer tier. Phases 5–7 build on top of it. Phase 4 is where you cross the threshold from “applied ML at Zoho” to “inference engineer, credentialed by public work.”

Ship these five things. They are the phase.