Rung 7 — The Enterprise Reference Architecture¶

Aligned phase: Phase 7 (production + reliability) Ship by: end of M12 Effort: ~40 hrs of writing + ~20 hrs of measurement on your rung-5 engine and vLLM. Signal: staff-engineer-tier. This is the document that makes a customer’s CTO forward your name to their VP. It is the artifact that turns “good engineer” into “engineer who can be trusted with a $2M procurement.”

Why this rung is staff-engineer-tier¶

The previous six rungs prove you can build. This rung proves you can prescribe at scale. It is the difference between “I optimized a kernel” and “I sized, deployed, and operated a fleet.” Hiring committees for staff+ roles at Anthropic, Databricks, Together AI, Anyscale, Fireworks, and every enterprise ML platform team are looking for one document like this on your GitHub. Ninety percent of applicants can’t produce it. That is your edge.

It is also the artifact that turns Zoho from “where you work” into “who you speak for.” Your day job gives you access to workload data nobody outside the company has. Sanitized, generalized, published — that workload knowledge becomes your public credential.

The artifact spec¶

A 15–25-page document (sanitized public gist + a private internal doc), structured as a real deployment RFP would demand. Reference build: 08_production/12_reference_architecture_capstone.md. That file has the measured numbers, the hardware bill, the SLO commitments. This file tells you how to turn it into a portfolio artifact that reads like something Databricks Solutions or Anthropic Applied would ship.

The scenario — do not rewrite it, use it verbatim:

5000-seat on-prem CRM Copilot for an enterprise customer. Air-gapped. Data cannot leave the building. Two workloads: agentic multi-turn chat (100 concurrent, p95 TTFT < 800ms, p95 ITL < 60ms), and utility (classification/extraction, 1000 req/min, p95 e2e < 2s). One 8×H100 SXM node, 30kW rack, dual 100GbE. Existing customer traffic pattern: 75–90% prefix-cache hit rate on agent traces.

That scenario is not hypothetical. It is the scenario Zoho and every peer CRM vendor is quoting on right now. That is why writing it up is high-leverage.

The 8-section structure — write in this order, do not reorder¶

1. Workload characterization (2 pages)¶

Two workloads defined by first principles: what messages arrive, how big, how often, how correlated (prefix reuse).
Table of arrival stats: mean/p95 request length, mean/p95 output length, mean/p95 concurrent sessions, prefix reuse ratio.
The one number that carries the section: 75–90% prefix-cache hit rate on Turn 2+ of agent traces. Cite it. Everything downstream is designed around it.
Zoho angle: “Measured on my team’s agentic harness traces. The shape generalizes; the exact numbers are internal.”

2. Hardware bill (1 page)¶

1 node, 8×H100 SXM 80GB NVLink, 2TB DDR5, 4×3.84TB NVMe RAID10, dual 100GbE, 30kW rack, redundant PSUs, ECC-DBE alerting configured.
BOM with rough prices (Supermicro / Dell reference SKUs). Do not quote your employer’s contracts.
One sentence on why NVLink and not PCIe-only: TP=4 requires NVLink for AllReduce latency to stay under a few hundred microseconds at 70B scale.

3. Model + quant + engine selection (2 pages)¶

Chat: Llama-3.3-70B-Instruct at FP8. Justify with the rung-4 bake-off numbers.
Utility: Qwen2.5-7B-Instruct or Phi-4-14B at FP8 or W4A16 AWQ. Justify with cost-per-1M-tokens.
Engine: vLLM. State the SGLang comparison honestly: “SGLang wins on RadixAttention semantics; vLLM wins on ops maturity, release cadence, and enterprise deployment tooling. For a 5000-seat production install with a 3-year support horizon, ops maturity is the tiebreaker.”
Router: SGLang Router or llm-d gateway for prefix-cache-aware routing across replicas. Not round-robin — that erases the 75–90% cache hit rate.
One prescription per line. No “depending on.”

4. SLO commitments (1 page)¶

The table:

| Workload | Metric        | Target                      | Alarm at | Measured   |
|----------|---------------|-----------------------------|----------|------------|
| Chat     | p95 TTFT warm | < 800 ms                    | 1000 ms  | ~250 ms    |
| Chat     | p95 TTFT cold | < 1500 ms                   | 2000 ms  | ~750 ms    |
| Chat     | p95 ITL       | < 60 ms                     | 80 ms    | ~45 ms     |
| Chat     | Concurrent    | 100                         | drops>2% | 100 sust.  |
| Utility  | p95 e2e       | < 2 s                       | 3 s      | measured   |
| Utility  | Throughput    | 1000 req/min                | <900     | measured   |
| Cluster  | KV util       | < 90% steady, <95% burst    | 95%      | 65–80%     |
| Cluster  | Prefix hit    | > 70% steady on agent load  | <60%     | 75–90%     |

Every row has a target, an alarm threshold, and a measured number. Rows without measured numbers are wishes, not commitments.

5. Capacity plan (2 pages)¶

Partition: 2× TP=4 replicas of Chat 70B on the two NVLink domains A(GPUs 0–3) / B(GPUs 4–7). Utility as LoRA over the same 70B base (Layout A) or as a separate small model on shared GPUs (Layout B). Prescribe Layout A.
Per-replica measured budget: ~1500 tok/s aggregate, ~35 concurrent chat sessions at target latency, KV pool sized for 8k context × 40 sessions.
Headroom: run each replica at 70% steady-state to leave capacity for burst + failure of the peer replica. State the failover math: if replica A dies, replica B must absorb 200 concurrent sessions, which exceeds SLO. The plan is: degraded-mode SLO at 1200ms p95 TTFT during single-replica failure, restored on replacement. Name the degraded SLO explicitly.
Cold-start budget: 90 seconds from kubectl apply to first token, dominated by weight load from NVMe.

6. Observability (2 pages)¶

The stack in one sentence: Prometheus for metrics, Grafana for dashboards, Loki for logs, Tempo for traces, single-node K8s for lifecycle. No SaaS — air-gapped rules that out.

Dashboards (name each, one line on what it shows):

Fleet overview — request rate, latency percentiles, error rate, per-replica health.
Latency deep-dive — TTFT vs ITL vs queue-wait split, per replica.
KV cache — free blocks, allocated, prefix-hit rate, eviction rate, fragmentation.
GPU utilization — SM util, HBM util, HBM bandwidth, power draw, temperature, ECC error counters.
Cost + SLO — dollars-per-1M-tokens, SLO burn rate over rolling 7d.

Alerts (name each, threshold, page or ticket):

p95 TTFT > 1s for 5m rolling — page.
Replica health check failing > 30s — page.
KV utilization > 95% for 60s — page.
ECC DBE counter incremented — page immediately, hardware retirement path.
Prefix hit rate < 60% for 15m — ticket (router misconfiguration signal).
HBM bandwidth < 80% of expected during load — ticket (SM-clock or thermal signal).

7. Cost model (1 page)¶

Node all-in: ~$4.60/hr amortized (3-year hardware amort + power + cooling + ops overhead + license). Show the arithmetic. Do not cite marketing.
Chat: ~$0.42 / 1M tokens. Utility: ~$0.15 / 1M tokens.
Compared to GPT-4.1-class API: 5–15× cheaper depending on workload mix. The multiplier is your headline number.
Break-even math: at 5000 seats × moderate usage, break-even vs API is ~4–6 months. State it.

8. Runbook drills (2 pages)¶

Seven scripted failure-mode drills, run monthly:

Kill a replica during peak load. Verify router re-routes, degraded SLO honored, alerting fires.
Fill KV cache to 95%. Verify preemption + swap-out behavior, no OOM, no dropped connections.
Long-prompt DoS. 32k-context prompt at 10 req/s from one tenant. Verify chunked prefill + per-tenant rate limits contain blast radius.
Corrupted model signature. Startup must fail loudly, not silently serve a bad model.
Model-load-but-not-ready race. Health check must gate router registration.
GPU ECC double-bit error. Kubernetes cordons the node, workload drains to peer, retirement ticket auto-filed.
Two-week soak. Weekly automated ShareGPT-like replay at production QPS. Regression detection on p95 latency and prefix hit rate.

Each drill has an expected outcome, a red flag list, and a post-mortem template. Publishing the drills is what makes this document staff-tier. Anyone can list SLOs; naming the drills that verify them is the rarer skill.

Deliverables list (put this in the doc’s front matter)¶

Sizing memo (this document, sections 1–5).
Runbook (section 8 as an operations guide).
Deployment playbook (Helm chart + K8s manifests + Ansible if pre-K8s).
Benchmark report (measured numbers reproducing the table in section 4).
Cost sheet (spreadsheet linked, arithmetic reproducible).
License compliance memo (per model: Llama community license, Qwen Apache-2.0, redistribution rights, weight-export controls).

Six deliverables. Not five, not seven. Six is what an enterprise procurement will actually ask for.

The high-level architecture diagram (put this in the doc)¶

One diagram, at the top of section 5. Not fifteen. If a reviewer can’t understand the deployment from that mermaid block, add a caption; do not add another diagram.

Publishing strategy — the two-doc split¶

The public sanitized gist:

Everything above with all Zoho-specific numbers replaced by ranges (“75–90% prefix hit rate on agentic multi-turn traces from a CRM assistant workload”).
Hardware bill with generic Supermicro SKU, no vendor pricing.
Cost model with the arithmetic template but placeholder rates.
Runbook drills verbatim — those are not proprietary.
URL: gist.github.com/<you>/enterprise-reference-arch-onprem-llm.md

The internal doc (do not publish):

Actual numbers, actual pricing, actual replay traces, actual customer names if applicable.
Kept in your team’s wiki. Cited in RFPs. This is what turns the artifact from portfolio into revenue.

Get manager sign-off on the sanitization before posting the gist. Not permission for the concept — permission for the specific paragraphs that reference workload shape. Ask in writing. Keep the reply. Every enterprise ML person you meet will ask about this doc, and you want the paper trail.

Where to post¶

Public gist (primary): gist.github.com/<you>/enterprise-reference-arch-onprem-llm.md. Long-form, one file, versioned, quotable, cite-able. This format works for this artifact where a full repo does not.
Personal blog: cross-post as “Sizing an on-prem LLM deployment for 5000 seats, air-gapped.” This is the highest-SEO-value title in your entire portfolio. Enterprise architects Google this exact phrase.
LinkedIn: yes, aggressively. This is the artifact LinkedIn was invented for. One post, 5 bullets, gist link. Enterprise buyers live on LinkedIn.
Hacker News: try once. If it lands, the traffic is huge. If it doesn’t, no harm.
X/Twitter: thread. Tag @vllm_project @sgl_project @NeuralMagic and one or two enterprise-inference voices (Chip Huyen, Simon Willison). Only if the numbers are defensible.
GPU MODE Discord #production: share once. Ask for review from someone who has actually run inference at scale.
r/LocalLLaMA: the on-prem angle plays well. Post title: “How I’d size an air-gapped 5000-seat LLM deployment on 8×H100. Numbers, arithmetic, runbook drills.”

Do not post to r/MachineLearning — wrong audience for a deployment doc.

What signals it sends¶

“I can operate at the SLO/capacity/cost planning level, not just the code level.”
“I have real workload data behind my prescriptions.”
“I write documents that a CTO can hand to a procurement lead.”
“I have opinions on multi-replica failover, license compliance, and drill cadence.”
“I can be trusted with a $2M hardware decision.”

That last bullet is the entire rung. A staff+ hiring committee sees this document and reads it as: this person can lead the deployment side of an infra team. That is a $50–100k/year compensation delta vs. “can write kernels.”

The Zoho angle (this is where it stops being subtle)¶

Rung 4 mentioned Zoho once. Rung 5 mentioned it once. Rung 7 is built on Zoho workload data. The scenario is the Zoho scenario. Own it.

Section 1 opens: “The workload shape here is measured from my team’s agentic harnesses at Zoho — CRM copilot flows, 3–6 turn tool-calling loops. Numbers generalized; no customer data.”
Section 4’s prefix-hit-rate row cites the measurement.
Section 7’s cost comparison references “a mid-market CRM SaaS pricing envelope,” not any specific SKU.

This is the artifact that lets a hiring conversation begin: “Tell me about Zoho’s inference stack.” You want that conversation to happen. Do not sanitize the domain out of existence.

Past examples to study (read all three before writing)¶

Anyscale’s Ray Serve reference architectures (blog series, 2024–25). The tone and structure for a production-grade prescription.
Databricks Model Serving whitepapers. The template for SLO tables and capacity math.
Meta’s Llama 3 tech report + serving-related engineering blogs. For “how a real team writes about a real deployment.”
NVIDIA’s Deployment guide for Llama-3-70B on H100 posts (there are several). Read the arithmetic in them, not the marketing.
The vllm-project/production-stack repo — read the K8s manifests and Grafana dashboards. Your runbook drills reference this exact stack.
The llm-d design docs (llm-d.ai / GitHub) — the disaggregated-serving reference the whole industry is aligning to.

Common mistakes¶

No numbers. “We recommend an appropriate number of replicas” is not a reference architecture; it is a shrug. Every claim has an arithmetic backing.
Round-robin router. The single most common mistake for on-prem LLM deployments. It erases prefix-cache hit rate and collapses SLOs. Prescribe prefix-cache-aware routing explicitly.
Single replica. No enterprise deployment ships with one replica. Even if utilization argues for one, HA argues for two. This is a non-negotiable.
No degraded-mode SLO. “We meet all SLOs” — until one replica dies. Publish the degraded numbers.
Cost model without arithmetic. “5–15× cheaper than API” is a marketing claim without the derivation. Show the tokens/sec, the dollars/hour, and the division.
Skipping the license memo. Llama community license has redistribution constraints; “open-source” is not the same as “you can bake it into a shipped product.” Enterprise procurement will ask.
Publishing internal numbers. One paragraph with actual tenant data ends the artifact and possibly the job. Sanitize twice, get sign-off, publish once.
Twenty diagrams. One high-level mermaid + inline tables > fifteen architecture diagrams. Density is credibility.
No runbook drills. SLOs without drills are aspirations. Drills are what turn architecture into operations.

Success signals (measure at 3–6 months)¶

Cited in an internal RFP response at Zoho or referenced in a customer conversation.
DM or email from an external enterprise architect asking about the drill list.
Referenced in a blog post or talk by someone at vLLM / SGLang / llm-d.
Recruiter outreach that quotes the document specifically (not the resume).
Featured on vllm-project’s community showcase or llm-d’s reference-deployment list.

If ≥2 hit, this document is doing its job. If a customer conversation cites it, this rung has paid for the entire ladder in one contract.

Next step¶

On the Monday after you publish this, you open the vLLM issue tracker (or SGLang, whichever you chose for rung 6), filter by the one subsystem your reference architecture leans on hardest — prefix caching, router policy, chunked prefill, KV eviction — and start tracking every issue in that area for two weeks. Do not open a PR. Do not comment. Read. Watch which maintainer merges what, which reviewer asks the sharpest questions, which corners of the code are undergoing change. That two-week reconnaissance is the entrance to rung 8. Rung 7 defined what enterprises need; rung 8 is you becoming the person who ships one piece of it upstream.

The ladder is the CV. Every rung is public. Every rung compounds.