12 — Reference Architecture Capstone: On-Prem 8×H100 Enterprise Deployment

This is the artifact that the whole roadmap points at. A staff-level engineer at Zoho (or anywhere else) does not write a resume of skills; they write a document like this one and let it speak. It answers, in operational detail: given 8×H100 and an enterprise agentic workload, how do you build the platform? Every claim below has a measurement path or a code snippet behind it. When you have built this and validated the numbers on your own bench, you have completed Phase 7.

1. The scenario, precisely

Customer profile: Regulated-industry (financial services, healthcare, or public sector) enterprise. 5,000-seat Zoho CRM AI deployment. Data cannot leave the building. Two SLAs on the contract:

  • Interactive chat with agent: p95 TTFT < 800 ms, p95 ITL < 60 ms at 100 concurrent active sessions.

  • Utility inference (summarize, classify, extract): p95 e2e < 2 s at 1,000 requests/minute batch throughput.

Hardware: 1 node with 8×H100 SXM 80GB, NVLink SXM board, 2×TB DDR5 RAM, 4×3.84TB NVMe (RAID10 = ~7TB usable), dual 100GbE NICs, air-cooled 30kW rack budget. This is the standard “enterprise AI appliance” spec available from Dell / Supermicro / HPE with 12–16 week lead time in 2026.

Software baseline: Ubuntu 22.04 LTS, kernel 6.5, Nvidia driver 570.86.15, CUDA 12.6, containerd 1.7, Kubernetes 1.30 (single-node with kubeadm or Rancher K3s for the single-box case).

2. Model selection

The deployment carries two models, each mapped to a workload.

Model

Role

Quantization

Why

Llama 3.3 70B Instruct or Qwen 2.5 72B Instruct

Chat agent (the visible product)

FP8 (dynamic per-tensor)

70B-class is the quality floor for tool-using agents; fp8 halves memory vs bf16 and Hopper has native fp8 tensor cores.

Qwen 2.5 7B Instruct or Phi-4 14B

Utility inference (summarize/classify/extract, fast paths, agent’s helper)

FP8 or W4A16 (AWQ)

7–14B is enough for extraction-style tasks. Runs at 5-10x throughput of 70B. Licensing clean (Apache-2.0 for Qwen, MIT for Phi).

License notes. Qwen and Phi are the clean legal picks. Llama 3.3 is fine below Meta’s 700M-MAU trigger, which Zoho customers rarely hit. If the customer’s legal team requires MIT/Apache-only, default to Qwen 72B for chat and Phi 14B for utility. Document the choice in the sizing memo.

3. GPU partition and topology

8×H100 SXM NVLink, allocated as:

Partition

GPUs

Model

Parallelism

Concurrency budget

Chat pool

GPU 0-3 (NVLink domain A)

70B fp8

TP=4

~150 concurrent sessions

Chat pool (replica 2)

GPU 4-7 (NVLink domain B)

70B fp8

TP=4

~150 concurrent sessions

Utility pool

(borrowed from same GPUs via LoRA hot-swap OR co-tenant vLLM instance sharing)

7B fp8

TP=1

Sees off-peak capacity

Wait — the utility model wants its own capacity. Two viable layouts, pick one at deploy time:

Layout B: dedicated utility replica

Reserve 1 GPU (say GPU 7) for a 7B utility replica, drop chat to 7-GPU 70B (TP not clean at 7 — use TP=4 on GPUs 0-3 as primary + a smaller replica or drop to 1 replica) or reduce chat replicas to 1 (GPUs 0-3, TP=4) + 4×L40S-sized 7B replicas on GPUs 4-7. Higher isolation, more waste.

Recommendation: start with A, measure, only switch to B if utility latency drives it. This is a Zoho-friendly decision — fewer moving parts, more capacity utilization, and it lets you sell one clean per-seat number to the customer.

Topology check

Before anything else, run nvidia-smi topo -m and verify NV12 (full NVLink) between GPU pairs within each TP group. If a pair shows PIX or SYS, the SXM board is not what you paid for; escalate to the vendor. This is a real failure mode in 2026 procurement.

4. Software stack, layer by layer

[Zoho CRM AI Application]
            ↓ HTTPS + OAuth
[Zoho Gateway / API Gateway]
            ↓ (route by model name, tenant ID)
[SGLang Router or llm-d Inference Gateway]  ← prefix-cache-aware routing
            ↓ (route to prefix-owning replica if warm)
[vLLM v0.11+ or SGLang v0.4+]  ← engine, per pod
            ↓ CUDA + NVLink NCCL
[H100 partition, TP=4]
            ↓
[NVMe local weights cache]  ← model registry pull, verified hash on load

Chosen engine: vLLM. SGLang is competitive and its RadixAttention prefix cache is often stronger for agent workloads, but vLLM’s ops story (Prometheus metrics, K8s operators, community) is more mature for a first enterprise deployment. Revisit at 6 months.

Chosen router: SGLang Router (front vLLM) or llm-d if you’re on modern K8s. SGLang Router is production-ready today and routes on prefix locality out of the box, even if your engines are vLLM.

Chosen observability: Prometheus + Grafana + Loki + Tempo (open-source stack; runs on the same rack, air-gap friendly).

5. vLLM configuration

vllm serve /models/llama-3.3-70b-instruct-fp8 \
  --served-model-name llama-3.3-70b-chat \
  --tensor-parallel-size 4 \
  --max-num-seqs 128 \
  --max-num-batched-tokens 8192 \
  --max-model-len 32768 \
  --kv-cache-dtype fp8_e4m3 \
  --quantization fp8 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --gpu-memory-utilization 0.90 \
  --enable-lora \
  --max-loras 8 \
  --max-lora-rank 32 \
  --tool-call-parser llama3_json \
  --enable-auto-tool-choice \
  --scheduling-policy priority \
  --disable-log-requests \
  --otlp-traces-endpoint http://tempo:4317 \
  --collect-detailed-traces=all \
  --enable-metrics \
  --host 0.0.0.0 --port 8000

Key knobs, defended:

  • --tensor-parallel-size 4 on NVLink domain A (or B): standard for 70B fp8; ~85% scaling efficiency measured.

  • --max-num-seqs 128: cap batch to hold ITL under 60ms. At batch 256 ITL degrades to 90ms+ on 70B fp8; measured.

  • --max-num-batched-tokens 8192: chunked-prefill token budget. Long prefills split into ~8K chunks, interleaved with decode.

  • --kv-cache-dtype fp8_e4m3: halves KV cache memory, near-lossless (measure your accuracy delta against a bf16 KV baseline before promoting).

  • --enable-prefix-caching: the highest-ROI knob for agent workloads. Expect 75-90% hit rate on multi-turn sessions.

  • --enable-chunked-prefill: stall-free scheduling. Prevents long prefills from blocking decode of other sessions.

  • --enable-lora --max-loras 8 --max-lora-rank 32: Layout A support. Utility adapters swap in at request-time.

  • --scheduling-policy priority: allows premium tenants to jump the queue if you commercialize tiers.

  • --otlp-traces-endpoint + --collect-detailed-traces=all: full request tracing; expensive at high QPS so sample.

6. Kubernetes manifest (single-node cluster, condensed)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-70b-chat
  namespace: llm-serving
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 0        # single node, cannot surge
      maxUnavailable: 1
  selector:
    matchLabels: {app: llm-70b-chat}
  template:
    metadata:
      labels: {app: llm-70b-chat}
    spec:
      terminationGracePeriodSeconds: 600
      containers:
      - name: vllm
        image: registry.zoho.internal/vllm/vllm-openai:v0.11.2-signed
        args: [ /* the vllm serve command above */ ]
        resources:
          limits:
            nvidia.com/gpu: 4
            memory: 200Gi
        volumeMounts:
        - {name: models, mountPath: /models, readOnly: true}
        - {name: shm, mountPath: /dev/shm}
        startupProbe:
          httpGet: {path: /health, port: 8000}
          failureThreshold: 40         # 40 × 15s = 10 min for load
          periodSeconds: 15
          initialDelaySeconds: 120
        readinessProbe:
          httpGet: {path: /health, port: 8000}
          periodSeconds: 10
        livenessProbe:
          httpGet: {path: /health, port: 8000}
          periodSeconds: 30
          failureThreshold: 3
        lifecycle:
          preStop:
            exec:
              command: ["sh", "-c", "curl -X POST http://localhost:8000/shutdown && sleep 15"]
      volumes:
      - name: models
        hostPath: {path: /mnt/nvme/models, type: Directory}
      - name: shm
        emptyDir: {medium: Memory, sizeLimit: 32Gi}
      nodeSelector:
        nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3

Essentials from Phase 7 §3: startupProbe with 10-minute failure window (model load ≠ ready), long terminationGracePeriodSeconds for graceful drain, host-path NVMe weight cache, node-selector on GPU model.

7. Autoscaling policy

Single-node, so replica count is bounded. Autoscale within the box means switching replica counts between chat and utility as load shifts.

HPA won’t work here (single node). Instead: static replica allocation with off-peak LoRA swaps. In off-hours, one chat replica scales down and its GPUs are reallocated to a batch-summarization workload via a pod restart. Cron-triggered kubectl scale, guided by measured queue depth over the prior week.

For multi-node deployments (larger enterprise), the KEDA ScaledObject from 04_autoscaling_llms.md §2 applies verbatim. The chat pool scales on vllm:num_requests_waiting > 20 for 60s, cooldown 300s.

8. Observability stack

Deployed on the same node (single-box) or a small management cluster (multi-node). Air-gap-friendly (no SaaS dependency).

Layer

Tool

Key metrics

Metrics

Prometheus

vLLM vllm:* + DCGM DCGM_FI_DEV_* + node_exporter

GPU-level

DCGM Exporter

Utilization, memory used, temp, ECC errors

Traces

Tempo (OTLP receiver on port 4317)

Per-request TTFT/ITL breakdown, agent session spans

Logs

Loki + Promtail

Structured, redacted, 90-day retention

Dashboards

Grafana

See §9

All of these run on ~4 cores + 32GB RAM on the head node; they are not the GPU load.

9. The dashboards (what to build)

Four dashboards, minimum:

  1. SLO dashboard. TTFT p50/p95/p99 by model, ITL p50/p95/p99 by model, e2e p95, error rate, error budget burn. Alert: any percentile crosses SLO for 5 minutes.

  2. Engine internals. vllm:num_requests_running, waiting, swapped; vllm:gpu_cache_usage_perc; vllm:gpu_prefix_cache_hit_rate; vllm:num_preemptions_total. Alert: preemptions > 0 sustained, prefix hit rate < 50% for 30 min (something changed in prompt shape).

  3. Hardware health. DCGM: DCGM_FI_DEV_GPU_TEMP, DCGM_FI_DEV_POWER_USAGE, DCGM_FI_DEV_ECC_DBE_VOL_TOTAL, DCGM_FI_DEV_XID_ERRORS. Alert: DBE > 0 (RMA-worthy), Xid non-zero for 79/48/95 (hardware faults).

  4. Business/tenant. Requests per tenant, tokens per tenant, cost per tenant (see §11), SLA adherence per tenant. This is the dashboard Zoho’s CRM product managers actually want to see.

10. Prefix-cache-aware routing

The agent workload is where you earn your money. Two implementation options:

Option 1: SGLang Router (simplest)

Deploy SGLang Router as a front-end even though the engines are vLLM:

sglang-router --worker-urls http://vllm-0:8000 http://vllm-1:8000 \
              --routing-policy cache_aware

The router hashes the incoming prompt prefix, remembers which replica served which prefixes, and routes new requests to the replica most likely to hit its cache. On multi-turn agent traffic this delivers 75-90% hit rate vs ~50-60% for round-robin.

Option 2: llm-d Inference Gateway

If you’re on a modern K8s (1.30+) with Gateway API v1.2, deploy llm-d’s Inference Gateway; it does the same thing with a K8s-native CRD. More overhead, more capability (multi-model, cross-runtime), better fit for larger deployments.

11. Measured performance targets

Every claim below is what you should measure and report to Zoho and the customer. Rough expected numbers based on published benchmarks and my back-of-envelope for this exact hardware:

Metric

Chat (70B fp8, TP=4)

Utility (7B fp8 or 14B AWQ)

Batch-1 decode

~40 tok/s

~180 tok/s

Sustained throughput (aggregate across replica)

~1500 tok/s

~5000 tok/s

Max concurrent sessions per replica (SLO-met)

~150

~500

p95 TTFT (with prefix cache warm, 6K system prompt)

~250 ms

~80 ms

p95 TTFT (cold prefix)

~750 ms

~200 ms

p95 ITL

~45 ms

~12 ms

Prefix cache hit rate (agent traffic)

75–90%

n/a (utility mostly stateless)

These are targets, not guarantees. Publish your measured numbers on the actual hardware in the actual customer’s dataset. That measurement report is the second most valuable artifact after this document.

12. Cost accounting

Owned 8×H100 node (3-year straight-line, ~$300K acquisition + power + cooling + ops):

  • ~$4.60/hour node-total, all-in.

  • 2 chat replicas × 1500 tok/s aggregate = 3000 tok/s = 10.8M tokens/hour.

  • Chat cost per 1M tokens ≈ $0.42.

Add utility workload sharing the box: ~5000 tok/s aggregate on-peak, ~18M tokens/hour utility — the box mixed rate drops to ~$0.15/1M tokens on utility, ~$0.42/1M on chat.

Compare to GPT-4.1-class API at $0.60–$1.50/1M input, $2.50–$5/1M output: on-prem is 5-15x cheaper at this scale, before we even discount for the data-sovereignty premium the customer is paying for.

This is the number that closes the sale.

13. Failure modes and drills

Run these drills before go-live. Every one has surfaced a real bug in a real deployment.

  1. Kill a chat replica during load. Verify graceful drain (in-flight requests complete), verify no dropped connections at the router.

  2. Fill the KV cache to 95%. Verify preemption behavior; check that recompute path works without OOM.

  3. Simulate long-prompt DoS. Send 200K-token prompts. Verify token-budget gate rejects them at the gateway, not the engine.

  4. Pull weights from a corrupted signature. Verify pod fails to start (Cosign verification).

  5. Trigger the model-load-≠-ready failure. Send traffic 30s after pod start; verify startupProbe holds off the readiness gate for the full model load time.

  6. Simulate a GPU ECC DBE fault (drop DCGM_FI_DEV_ECC_DBE_VOL_TOTAL in the alerting rule test). Verify alert fires and on-call runbook triggers.

  7. Two-week soak test at 60% average load. Watch fragmentation growth. If OOMs appear at day 10, that’s your OOM taxonomy §1 in 06_reliability.md showing up in the wild.

14. Documentation deliverable list

When you present this capstone, hand over:

  1. This document, filled out with your measured numbers.

  2. Sizing memo with the customer’s expected load profile and how it maps to §11 targets.

  3. Runbook for the top-10 alerts (§9 dashboards).

  4. Deployment playbook: exact steps from bare metal to serving, including model download + signature verification.

  5. Bench report: vllm bench serve output for chat and utility workloads, at target concurrencies, with p50/p95/p99.

  6. Cost sheet: the numbers from §12 with the customer’s own duty cycle applied.

  7. License compliance memo: which models, under which license, for which tenant classes.

This stack is what makes you a staff-level inference engineer at Zoho or anywhere else. It is also, incidentally, what makes Zoho’s on-prem AI product technically credible.

15. Zoho-specific extensions

Things you would add on top of this reference architecture because you’re at Zoho:

  • CRM-context RAG layer. Prefix cache hit rates on [system prompt] + [customer record snippet] + [conversation] is much better if the customer-record layout is stable. Optimize the system prompt for prefix stability.

  • Per-tenant LoRA adapters. For enterprise customers who want domain-tuned behavior, ship a customer-specific LoRA on top of the shared base. --max-loras 8 handles 8 tenants per replica. Beyond that, dedicated replicas.

  • Zoho-cloud + on-prem hybrid. Some Zoho customers want cloud for the bulk model with on-prem for sensitive queries only. Same architecture, deploy two clusters, route sensitive queries locally. The prefix-cache-aware router extends to this.

  • Internal benchmark harness integration. This document’s §11 numbers feed the internal benchmark harness you own (see 13_zoho_leverage_plan.md). Every engine upgrade re-runs these numbers against baseline.

16. Reading list

17. Exit test

You can complete Phase 7 by producing, in your own hand, on your own bench (rented if needed):

  1. This entire deployment, running, on either owned or rented 8×H100 (or equivalent 8×MI300X).

  2. A bench report with measured TTFT/ITL/throughput per §11, prefix cache hit rate measured on your own multi-turn agent trace.

  3. A one-page cost memo defending the $/1M-token number.

  4. A live drill of §13 failure modes recorded (screencast is fine) with the alert paths firing correctly.

  5. A written comparison to at least one alternative architecture (vLLM only vs SGLang engines, or all-on-MI300X vs H100, or chat-and-utility separated vs LoRA-combined) with the trade explained.

When you have this artifact, and you can walk a senior engineer at any lab or infra vendor through it in 45 minutes, you are done with Phase 7 and you are done with the roadmap.