14 — Phase 7 Projects (with acceptance criteria)

Every prior document in this Phase describes concepts and tools. This one describes what you build. Each project has an acceptance criterion — a concrete artifact and measurement that proves you finished. Skipping acceptance criteria is how Phase 7 becomes tourism instead of ownership. The projects are ordered so each builds on the previous; if you can only do a subset, do them in order and stop when you run out of time.

Estimated total effort: 200-300 hours over 4-8 months at 10-15 hrs/week. Compressible to 3 months at Zoho-work-aligned scope.

Project 1: Production vLLM Deployment on Kubernetes

Effort: 20-30 hours. Prereq: Phase 4 mini engine complete; access to at least 1×L40S or 1×H100 (rented is fine).

Build:

  1. Single-node K8s (kubeadm or K3s) with Nvidia GPU Operator installed.

  2. Deploy vLLM v0.11+ serving Qwen 2.5 7B fp8 with --enable-prefix-caching --enable-chunked-prefill.

  3. Startup / readiness / liveness probes correctly distinguishing model-load from serving-ready.

  4. Prometheus + Grafana scraping vLLM /metrics. Import a Grafana dashboard from vLLM’s docs; verify every panel populates.

  5. Deploy DCGM Exporter alongside; add GPU-level panels (utilization, memory, temp, ECC).

  6. Send load with vllm bench serve at target concurrencies 1, 8, 32, 128. Capture TTFT and ITL p95 per level.

Acceptance criterion:

  • A running deployment, screenshottable dashboard, and a Markdown report with the p95 TTFT/ITL curve as a function of concurrency. The report explains where the knee is and which knob you would turn to move it.

  • Screencast (5 min): rolling upgrade with zero dropped requests, then a kubectl delete pod that drains gracefully.

Project 2: Autoscaling that actually works

Effort: 15-25 hours. Prereq: Project 1.

Build:

  1. Add a second replica of the vLLM deployment.

  2. Install KEDA. Configure a ScaledObject with two Prometheus triggers:

    • vllm:num_requests_waiting > 10 (queue depth)

    • vllm:gpu_cache_usage_perc > 0.85 (KV pressure)

  3. minReplicas: 1, maxReplicas: 4, cooldownPeriod: 300s.

  4. Simulate load with a Poisson-arrival load generator (or a bash while-loop with xargs -P) that varies from 5 req/s baseline to 50 req/s peaks.

  5. Chart replica count vs queue depth vs TTFT p95 during the load pattern.

Acceptance criterion:

  • Chart shows KEDA scaling up during queue-depth spikes and scaling down after cooldown. TTFT p95 stays within your SLO through the transitions.

  • A companion “failure mode” experiment: replace the trigger with nvidia_smi_utilization_gpu > 0.80 (the wrong signal); demonstrate that scaling does not respond correctly to actual load. Blog-post-worthy.

Project 3: Prefix cache measurement on your own agent trace

Effort: 10-15 hours (mostly data prep). Prereq: Project 1.

Build:

  1. Generate a synthetic multi-turn agent trace: 100 conversations of 20 turns each, all starting with the same 4K-token system prompt (including tool definitions), each user turn adding 300-800 tokens.

  2. Replay the trace against vLLM with --enable-prefix-caching on.

  3. Replay against vLLM with --no-enable-prefix-caching.

  4. Measure: vllm:gpu_prefix_cache_hit_rate, prefill tokens processed total, aggregate throughput, mean TTFT.

Acceptance criterion:

  • Report showing hit rate (target >80%), prefill-tokens-processed reduction (target >4x), TTFT reduction (target 30-60% on warm turns).

  • This report is Zoho Play 1 dry-run material. Adapt it to real Zoho traffic when you can.

Project 4: Cost model calculator

Effort: 10-15 hours. Prereq: Projects 1-3.

Build: a spreadsheet or small Python script that:

  • Takes inputs: {model, quantization, hardware, GPU-hour price, target concurrency, target SLO}.

  • Outputs: predicted goodput (tokens/sec sustained under SLO), $/1M-token, break-even vs commercial API pricing (GPT-4.1, Claude, Gemini current rates), monthly cost at target QPS.

  • Cross-checks the predicted goodput against Project 1 measurements. Calibrate.

Acceptance criterion:

  • Calculator that produces the §12 numbers in 12_reference_architecture_capstone.md from spec inputs, within ±20% of measured.

  • Written analysis: for two Zoho product profiles (one interactive, one batch-y), pick the winning hardware + engine + quant tuple defensibly.

Project 5: OpenTelemetry tracing through an agent loop

Effort: 15-20 hours. Prereq: Project 1.

Build:

  1. Deploy Tempo alongside Prometheus + Grafana.

  2. Configure vLLM with --otlp-traces-endpoint http://tempo:4317 --collect-detailed-traces=all.

  3. Write a small agent harness in Python that calls the vLLM OpenAI endpoint multiple times in a loop with tool calls. Instrument the harness with OpenTelemetry SDK; propagate traceparent headers.

  4. Verify in the Tempo UI that a single agent session shows as one trace spanning all vLLM calls with the correct span hierarchy.

  5. Add agent-specific span attributes: agent.session_id, agent.turn_number, agent.tool_name, llm.prefix_cache_hit (extracted from vLLM response metadata).

Acceptance criterion:

  • A screenshot of a 5-turn agent session in Tempo, with the tool-call spans nested under LLM-call spans, showing per-span TTFT, ITL, and cache-hit status.

  • Explain what percentile of ITL is dominated by decode vs router overhead vs network latency, with numbers.

Project 6: Reliability drills

Effort: 10-15 hours. Prereq: Projects 1, 2.

Build: execute each drill from 12_reference_architecture_capstone.md §13:

  1. Pod kill during load.

  2. KV cache fill to preemption.

  3. Long-prompt DoS.

  4. Corrupted-signature model load rejected.

  5. Model-load-≠-ready readiness gate check.

  6. Simulated ECC DBE alert firing.

Record each outcome, alert firing, and recovery path in a runbook document.

Acceptance criterion:

  • A runbook with 6 drill entries; each entry has: what was injected, what the metric/alert showed, what the operator response was, what changed to prevent recurrence.

  • This runbook is directly transferable to Zoho or any customer engagement.

Project 7: Model registry + signing pipeline

Effort: 20-30 hours. Prereq: none, though projects 1-6 provide context.

Build:

  1. Deploy MinIO (or use an existing S3-compatible bucket).

  2. Postgres for metadata: model name, version, upstream URL, upstream hash, ingestion timestamp, license class, signature, ingester identity.

  3. Ingestion script: hf download + verify hash + compute SHA-256 + cosign sign-blob + upload to MinIO + insert Postgres row.

  4. Verification script: given a model name + version, download from MinIO + verify Cosign signature + verify hash + load into vLLM.

  5. Rejection path: an intentionally corrupted upload; verify pipeline blocks it.

Acceptance criterion:

  • Working registry with at least three ingested models (one Apache, one MIT, one Meta-Community-License, tagged accordingly).

  • Written procedure document that a colleague could follow to ingest a new model.

  • This is the direct backbone of Zoho Play 3.

Project 8: Fine-tune a 7B model and beat a commercial API on a bounded task

Effort: 40-60 hours. Prereq: Phase 5-6 fine-tuning experience; access to 1-2×H100 for a day, or a 24-hour spot instance.

Build:

  1. Pick a bounded task: e.g., extract structured fields (customer name, priority, due date, category) from a natural-language support ticket. Synthesize a training dataset of 2000-5000 examples using a stronger model.

  2. Fine-tune Qwen 2.5 7B with LoRA (rank 32). Use unsloth or axolotl for speed. Total compute cost < $100.

  3. Evaluate on a held-out test set: field-level accuracy vs GPT-4.1-class baseline. Publish the confusion.

  4. Deploy fine-tuned model on vLLM with --enable-lora. Measure end-to-end latency and cost per 1000 tickets vs the API baseline.

Acceptance criterion:

  • Report showing quality (target: within 2% of GPT-4.1 on target task) and cost (target: 10-50x cheaper per 1000 requests) with methodology and full eval breakdown.

  • Direct backbone of Zoho Play 4.

Project 9: The Capstone (Reference Architecture bench)

Effort: 40-80 hours over 4-6 weeks. Prereq: Projects 1-6 minimum.

Build: the full architecture described in 12_reference_architecture_capstone.md. Since 8×H100 rental is expensive ($20-25/hr), execute it smaller first: 2×L40S or 1×H100 with a smaller model (Qwen 14B on 1×H100, or Qwen 7B on L40S). Once the topology and observability work, upgrade to 4×H100 or 8×H100 for one final measurement pass (budget ~$300-600 for the final run).

Acceptance criterion:

  • Full deployment running, with the four dashboards from 12_reference_architecture_capstone.md §9 populated with real numbers.

  • Bench report matching §11 targets within reasonable tolerance for the smaller hardware.

  • The seven documentation deliverables from §14 of that doc.

  • Written comparison to at least one alternative architecture.

This is the artifact that closes Phase 7 and closes the roadmap.

Project 10: OSS contribution to an inference engine

Effort: 20-100 hours depending on ambition. Prereq: deep familiarity with at least one engine’s source.

Build: land a merged PR to vLLM, SGLang, FlashInfer, or llama.cpp. Ideas that align with Phase 7 work:

  • Improve a Prometheus metric name, label, or histogram bucketing.

  • Fix a documentation bug in the deployment or observability sections (low-friction entry).

  • Add a benchmark script for an underrepresented workload profile.

  • Improve error handling for the OOM / preemption paths.

  • Add a serialization format audit tool.

Start in the project’s Slack/Discord; find a good-first-issue; take one; ship it.

Acceptance criterion:

  • One merged PR with your name on it, in a repo people cite. This is the credential in this field.

Project priority if time-constrained

If you can only do a subset (real life), do them in this order:

  1. Project 1 (production deployment) — mandatory.

  2. Project 3 (prefix cache measurement) — highest Zoho ROI.

  3. Project 4 (cost model) — unlocks every internal conversation.

  4. Project 2 (autoscaling) — mandatory for the “platform engineer” identity.

  5. Project 5 (tracing) — makes the debugging story real.

  6. Project 7 (registry) — backbone of Play 3.

  7. Project 8 (fine-tune) — backbone of Play 4.

  8. Project 6 (drills) — promotes you from “builder” to “operator.”

  9. Project 9 (capstone) — the crown, do it when the rest is in place.

  10. Project 10 (OSS PR) — do this in parallel with Phase 6, not just Phase 7.

Exit test for Phase 7 as a whole

You have completed Phase 7 when:

  1. Projects 1-4 are all complete with published artifacts.

  2. At least one of Projects 5-8 is complete.

  3. Either Project 9 is complete, or you have taken at least the equivalent Zoho work-project through the same acceptance criteria.

  4. You have shipped at least one merged OSS PR (Project 10).

  5. You can walk another engineer through the full stack from HTTP request to tensor core to Prometheus dashboard without notes.

When those five conditions hold: the roadmap is done. Congratulations. Now you get to actually do the job.