Phase 7 — Production & Enterprise-Grade Inference¶

Months 16–24 (and forever). This is the phase where you stop being “someone who understands inference” and become “someone who runs inference platforms.” It fuses everything you’ve learned into the profile that makes you hard to hire — and impossible to replace at Zoho.

Why this phase is your unfair advantage¶

Every other phase teaches you skills that thousands of ML engineers are also learning: transformer internals, CUDA kernels, quantization, distributed training. Phase 7 is where your Zoho background stops being background and starts being leverage.

You already have three things that kernel-first people don’t:

Production-services instincts. SLOs, on-call, graceful degradation, blast-radius thinking, capacity planning. You didn’t learn these from a paper; you got them the expensive way — by keeping long-running services alive.
On-prem deployment exposure. The single hardest inference market is enterprise on-prem, and it’s exactly where Zoho already lives. Data residency, air-gap, license vetting, hardware sizing under customer constraints — you’ve been adjacent to these problems for years.
Agentic workload literacy. You build harnesses. You know what a 12-turn tool-calling loop looks like. You know the token distributions. This is not generic knowledge; agentic workloads have wildly different economics from chatbot workloads (see 05_observability.md and 13_zoho_leverage_plan.md), and being able to reason about them concretely is a differentiator.

The rest of this phase is about making those three latent advantages cash out.

The mental frame: inference is now a distributed systems problem¶

The paradox of Phase 7 is that after 15 months of kernel-first study, the biggest wins are almost never kernel wins. At the platform level:

A prefix cache hit saves 10,000× more work than a 10% kernel speedup, because the entire prefill is skipped.
A wrong autoscaling metric (GPU util instead of queue depth) burns more money in a week than a quarter of quant work saves.
A model-load-≠-ready liveness probe bug takes down more prod traffic than any race condition in an attention kernel.
A 220 GB model image without registry caching turns a 30 second rolling upgrade into a 12-minute one — and blows every SLO for the customer.

Phase 7 is where you learn to see the whole stack — request arrives at Envoy, gets routed by a KV-cache-aware gateway, hits a vLLM pod on an H100 that just got scheduled with the right MIG partition, streams SSE back through Kubernetes, and shows up as a p95 TTFT number on a Grafana panel that alerts if it breaches the SLO. Everything you learned in Phases 0–6 is a leaf node in this tree. If any leaf is broken, the whole thing collapses. But the shape of the tree is what you now have to master.

Reading order and pacing (Months 16–24)¶

Do them in the sequence below. Each doc has a specific reason to come after the previous one; don’t skip forward.

#	Doc	Why it’s here
01	`01_engines_as_products.md`	The engines you already know (vLLM/SGLang) treated as products with a config surface, plus the orchestration layer (Dynamo, Triton, KServe, Ray Serve, llm-d).
02	`02_openai_api_spec.md`	The API contract everything speaks. Get this right or every agent breaks.
03	`03_kubernetes_for_gpus.md`	Where engines actually run. MIG vs time-slicing, GPU operator, the model-load-≠-ready gotcha.
04	`04_autoscaling_llms.md`	Why HPA fails and what to do instead.
05	`05_observability.md`	You cannot fix what you cannot see. OpenTelemetry + Prometheus + percentiles discipline.
06	`06_reliability.md`	OOMs, preemption, backpressure, multi-tenant isolation, abuse.
07	`07_on_prem_enterprise.md`	The Zoho manifesto. Air-gap, license vetting, data residency.
08	`08_sizing_exercises.md`	Napkin math with worked examples (“customer has 4×L40S — what do I promise?”).
09	`09_cost_modeling.md`	$/1M-tokens the honest way. Batch vs interactive tiers.
10	`10_gpu_procurement.md`	H100/H200/B200/B300/GB200/MI300X/MI325X/L40S/RTX Pro 6000 — 2026 prices and positioning.
11	`11_security_supply_chain.md`	Prompt injection, safetensors audit, supply chain.
12	`12_reference_architecture_capstone.md`	The staff-level artifact. On-prem 8×H100 serving 70B + 8B with prefix-cache-aware routing.
13	`13_zoho_leverage_plan.md`	Read this last. How every doc above pays you back at work.
14	`14_projects.md`	Acceptance-criteria-bound projects for this phase.

Phase 7 exit criterion (verbatim from the roadmap)¶

You can take a vague enterprise requirement (“we need Copilot-like features for 5,000 seats, data can’t leave the building”) and produce, unaided, a hardware bill, model/quant/engine selection, SLO commitments, and a capacity plan — each line defensible with arithmetic or a benchmark you know how to run.

If, after Month 24, you cannot walk into a Zoho architecture review and defend every line of a capacity plan against a hostile CTO with a calculator, you have not finished this phase. The capstone document in 12_reference_architecture_capstone.md is designed to force this competence.

The one thing to keep saying to yourself¶

Every layer of the stack has an SLO, and every SLO is somebody’s job. Phase 7 is where you make yourself the person whose job is all of them.

That person is rare, senior, and expensive. That’s the target.