06 — Reliability: OOMs, Preemption, Backpressure, Multi-Tenancy, Abuse

Latency is what customers feel on a good day. Reliability is what they remember on a bad one. In LLM serving the failure modes are unlike anything in traditional web serving: memory is dynamic and pathological, cost of failure is asymmetric (a single 200GB weight reload costs minutes), and the load shape is adversarial in ways your CRUD services never had to think about. This doc catalogues the failure taxonomy and the operational patterns that keep an inference fleet up when the traffic is real.

---

1. The OOM Taxonomy — Why Your GPU Died

“CUDA out of memory” is not a single failure. It is at least five different failure modes with different fixes. Getting the taxonomy right is the first reliability skill.

1.1 The five OOMs

#	Class	When	Root cause	Fix
1	Weights OOM	At startup	Model + framework overhead > VRAM	Smaller model / more quant / more TP / bigger GPU
2	KV growth OOM	Steady state, long sessions	Concurrent seqs × seq_len × KV bytes exceeded free pool	Cap max-num-seqs / max-model-len; enable prefix cache; enable swap
3	Fragmentation OOM	Hours in, under load	PyTorch caching allocator holes; long-lived tensors block free blocks	expandable_segments:True; upgrade PyTorch; periodic drain
4	Activation-spike OOM	On a specific request	A single very-long prefill blows past reserved pool	Enable chunked-prefill; set --long-prefill-token-threshold; reject very long prompts
5	Companion-process OOM	Anywhere	Tokenizer worker / logging sidecar / metrics scraper spikes host RAM, gets OOM-killed, takes engine with it	Pod memory limits + reservations; separate sidecars

The engine’s Prometheus metrics tell you which one before you read the traceback. vllm:num_preemptions_total rising steadily = class 2 (KV growth pushing scheduler into evict mode). Sudden preemption with gpu_cache_usage_perc still moderate = class 3 (fragmentation). Latency spike + one request cancelled + no others affected = class 4.

1.2 The reservation model in your head

vLLM at startup allocates roughly gpu-memory-utilization × total_vram and hands the rest to PyTorch’s caching allocator and CUDA context. Of the vLLM slab:

vram_vllm = weights + activations_peak + kv_cache_pool
kv_cache_pool = vram_vllm - weights - activations_peak
max_concurrent_tokens = kv_cache_pool / bytes_per_token_of_kv

For Llama-70B fp16 on 4×H100 SXM (4×80GB = 320GB, 90% util = 288GB usable):

• Weights: 140GB (35GB per shard with TP=4)

• Activations reserved (--max-num-batched-tokens 8192): ~4GB per GPU

• Remaining KV pool per GPU: ~40GB → ~160GB cluster-wide

• Per-token KV (Llama-70B, GQA 8 KV heads): 2 × 80 layers × 8 KV heads × 128 head_dim × 2 bytes = 320KB/token

• Max total resident tokens ≈ 160GB / 320KB ≈ 500,000 tokens across all concurrent sequences

If someone with a 128k-token prompt walks in and 3 concurrent 128k sessions already exist, you are one long-prompt away from class 4. Set --max-model-len to what your KV pool can survive at your realistic concurrency, not to what the model architecture supports.

1.3 The single most useful setting

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

This changes PyTorch’s caching allocator to a growable-segment strategy that dramatically reduces class 3 fragmentation. Every serious vLLM/SGLang production deployment sets it. Confirm it stuck by grepping the pod env — this is one of the top-3 debugging shortcuts.

1.4 The Zoho angle

Your on-prem customers cannot tolerate a fleet-wide OOM cascade at 3am. Ship each deployment with an OOM runbook that maps the five classes to the specific vLLM flag to change, and validate the runbook by causing each failure in staging (long-prompt torture test, 200-concurrent-session soak, 48-hour fragmentation soak). A vendor who has proved they can survive each failure mode on the customer’s hardware is a vendor who gets renewed. Almost no one ships this. You will.

---

2. Preemption vs Swap vs Recompute

When the KV pool saturates, the scheduler has to make room. Three strategies exist; each has a completely different latency profile. Understanding which one your engine chose matters for postmortems.

2.1 The three moves

Move	Mechanism	Cost	When it wins
Preempt (recompute)	Evict a sequence’s KV; put it back in queue; re-prefill from scratch when re-scheduled	Wastes prefill FLOPs; TTFT-equivalent spike on the victim	Short sequences, cheap prefill; default in vLLM V1
Swap (offload)	Copy the KV to host RAM; bring back when GPU has room	PCIe bandwidth spike per swap (~64 GB/s Gen4, ~128 GB/s Gen5); wall-clock latency for the victim	Long sequences where recompute is more expensive than a PCIe round-trip
Chunked-prefill co-scheduling	Never lets prefill starve decode; no preemption needed under steady load	None on hit; requires token-budget tuning	Prevention beats cure; default on in vLLM/SGLang since 2025

vLLM’s V1 scheduler defaults to recompute (preemption) because with prefix caching hot, re-prefill often costs less than a swap round-trip. Watch vllm:num_preemptions_total. A healthy production fleet has this metric slowly ticking, not spiking. Sustained preemption rate > 1 per second per replica = you are undersized or your max-num-seqs is set too aggressively.

2.2 The sneaky third failure

Class-3-fragmentation-triggered preemption looks identical to class-2-capacity preemption in the metrics but is a completely different problem. If your gpu_cache_usage_perc is at 60% and you’re still preempting, your allocator is fragmented, not full. That is when expandable_segments:True earns its keep.

---

3. Backpressure & Timeout Design

The default failure mode of a naive LLM server under load is: accept every request, queue infinitely, TTFT drifts to minutes, clients time out, retry, doubling the queue. This is the LLM-serving equivalent of a metastable failure. You must design the pressure valve deliberately.

3.1 The layers where you push back

Four places to reject or defer, in order of preference (earlier = cheaper):

1. Gateway rate limit (per API key, per tenant, per model). Cheapest reject; never touches the engine. Envoy/Kong/APISIX or your gateway of choice.

2. Admission control at the router (queue-depth aware). If vllm:num_requests_waiting on the target replica > threshold, return 429 with Retry-After before dispatching. Both llm-d Inference Gateway and Dynamo router do this natively.

3. Engine-level cap. vLLM’s --max-num-seqs and --max-num-batched-tokens cap what the scheduler will accept per iteration. New requests wait; the wait metric surges; autoscaler reacts.

4. Per-request timeout at the API layer. If the request has been queued longer than the client’s expected TTFT SLO, kill it before it ever prefills — you save the compute and the client got their 504 either way.

3.2 The retry-storm inequality

For any queue-based system:

if (retry_rate × retry_count > drain_rate): the queue is a bomb

LLM serving has a specific twist: retry cost equals full re-prefill on the retry replica, and if prefix-cache-aware routing is not wired up, the retry goes to a random replica whose cache is cold. Two mitigations that work in production:

• Client-side exponential backoff with jitter, mandatory. Publish this in your SDK and reject retries with no Retry-After header respect.

• Idempotency keys. If the client resends with the same idempotency key inside the window, the gateway can return the cached in-flight completion instead of dispatching a new one. Non-trivial; matters when you have flaky mobile clients.

3.3 Deadline propagation

Every internal call in the chain (gateway → router → engine) should carry a deadline header (x-deadline or gRPC deadlines). If the deadline has passed by the time the request reaches the engine, drop it. This is standard SRE for RPC systems and almost nobody does it for LLM serving. Do it and your tail latency under load looks like nobody else’s.

---

4. Multi-Tenancy: Fair Scheduling & Per-Tenant Budgets

The moment your platform serves more than one customer, “one greedy tenant kills everyone” becomes a Sunday-morning incident. The engines have partial support; the rest is on you.

4.1 Priority classes

vLLM supports request priority via the priority field (integer, higher = more important) and the scheduler will honour it when --scheduling-policy priority is set. SGLang has similar. Use it to build tiers:

• Interactive (priority 100): user-facing chat, low TTFT SLO

• Agentic (priority 50): tool-calling loops, medium latency, high token budget per session

• Batch (priority 10): async analytics, bulk summarization, embed-then-index jobs

Priority does not solve fairness on its own — a single interactive tenant flooding requests still starves the others.

4.2 Token buckets per tenant

The correct primitive is a token-bucket rate limiter keyed on tenant, applied at the gateway. Two dimensions:

• RPS bucket (requests per second) — cheap, coarse

• TPM bucket (tokens per minute, weighted by prompt + expected output) — the honest budget

Redis with a Lua script does this at low latency. OpenAI publishes their per-model limits publicly; that is the schema to imitate. For Zoho this maps cleanly to per-org limits already flowing through your API gateway — extending it with a token dimension is a week of work with outsized payback.

4.3 KV-cache fairness

Priority + rate limit is not enough. A tenant with a very long prompt hogs KV pool disproportionately. Add:

• Per-tenant max-tokens caps at admission

• Per-tenant concurrency caps (max in-flight sequences)

• KV budget accounting — an experimental feature in llm-d, watch the community for it. Until it lands, an approximation: multiply tenant priority by 1 / (avg_prompt_tokens_last_hour) in your custom router.

4.4 Isolation strategies (ranked by strength)

Strength	Mechanism	When
Weak	Rate limits + priority queues on shared engine	Most SaaS multi-tenancy
Medium	LoRA-per-tenant on shared base model	Fine-tuned tenants sharing a base
Strong	Dedicated replica per tenant	Big enterprise contracts, on-prem
Nuclear	Dedicated GPU / MIG slice / node	Regulated tenants, compliance mandate

For Zoho, the middle two are the money zone: LoRA-per-CRM-account on a shared Qwen or Llama base gives you strong-enough tenant separation and 10x the density. Dedicated replicas are how enterprise tiers get sold.

---

5. Abuse Patterns You Will See In Week 2

The moment your endpoint is exposed, adversarial and semi-adversarial traffic starts arriving. Some patterns:

5.1 The long-prompt DoS

An attacker sends max_model_len - 1 tokens of gibberish. Prefill is O(N²) in attention; a single request can stall your decode iterations for seconds. Multiply by 100 concurrent attackers and your interactive fleet is dead.

Mitigations, in order:

1. Reject at gateway on prompt_tokens > tenant_cap. Default cap = 16k unless raised.

2. Enable --enable-chunked-prefill (default on recent vLLM) so long prefills interleave with decode.

3. Set --long-prefill-token-threshold so the scheduler chunks aggressively.

4. Per-tenant prompt-token TPM bucket.

5. Reject prompts whose measured entropy is too low (gibberish detection); the mirror-nginx blocklist is a starting point.

5.2 The generation-length DoS

max_tokens=8192 on every request even for a simple factoid. Wastes decode bandwidth. Cap it: per-tenant, per-model, and enforce at the router before the engine sees it.

5.3 The tool-loop bomb

An adversarial agent that keeps calling tools in a loop, growing the conversation, consuming your context budget and burning tokens. Because agentic products at Zoho are the target here, this is the pattern to instrument for. Detect: agent.turn_number > threshold, or session_tokens_lifetime > threshold. Break out with a hard cap.

5.4 Prompt injection as reliability (not just security)

Prompt injection is treated in 11_security_supply_chain.md, but from a reliability lens: successful injections often manifest as tool-calling loops, malformed structured outputs, and unusually long generations. Alarm on all three and you catch injection attempts before someone else does.

5.5 The scraper

Same client key, n streaming requests with stream=true opened and never read. Server sockets pile up. Fix:

• Aggressive keepalive_timeout and keepalive_requests at the fronting proxy

• Server-side timeout on cumulative SSE-idle-time (Envoy stream_idle_timeout: 60s)

• Per-tenant concurrent-stream limit at the gateway

5.6 The prefix-poisoning attack (novel)

Because prefix caching is content-addressed, a hostile tenant on a shared engine can prefix-warm the cache with attacker-chosen strings, then observe timing to infer other tenants’ prompts. Real risk in shared multi-tenant, publicly reported. Mitigations:

• Prefix cache scoping: bind cache entries to tenant_id prefix. vLLM has a --enable-prefix-caching-hash-input mechanism; SGLang has a per-user cache namespace. Turn one of them on.

• Consider: no shared prefix cache across tenant boundaries at all (accept the cache-hit hit).

---

6. The Reliability Instrumentation Kit

The minimum set of alerts a serious platform ships with:

Alert	Signal	Threshold	Runbook link
Queue growing unbounded	rate(vllm:num_requests_waiting[5m]) > 0 for 15m	Sustained	Scale up / rate-limit / investigate hot tenant
Preemption storm	rate(vllm:num_preemptions_total[5m]) > 1/s	5 minutes	Class-2 or class-3 OOM diagnosis
TTFT SLO burn	histogram_quantile(0.95, vllm:time_to_first_token_seconds) > SLO for 5m	Sustained	Capacity investigation
ITL SLO burn	Same for time_per_output_token	Sustained	Batch too large, or KV pool starved
Preemption + high cache usage	Combined	Any	Class-2 (KV capacity)
Preemption + moderate cache usage	Combined	Any	Class-3 (fragmentation)
GPU ECC DBE	DCGM_FI_DEV_ECC_DBE_VOL_TOTAL > 0	Any	Cordon node, RMA GPU
GPU thermal throttle	DCGM_FI_DEV_CLOCKS_EVENT_REASONS ≠ 0	Any	Datacenter cooling issue
Xid errors	DCGM_FI_DEV_XID_ERRORS ≠ 0	Any	Kernel-level GPU fault; cordon and reboot
Model-load-≠-ready	Container Ready before /health/started	Startup	Fix probes per doc 03

Each alert should link to a runbook file with: symptoms, likely causes (ranked), one-command diagnostics, and the specific vLLM/engine flag or K8s action to take. The runbook set is a real deliverable — publish it internally at Zoho, and it becomes your reliability credibility artefact.

---

7. The Uncomfortable Truth About Long Runs

LLM engines are not yet mature enough to run for months without incident. Every serious production deployment restarts replicas on a schedule (weekly is common) to reset the caching allocator, the CUDA context, and the Python heap. This is stigma-free — it is called “planned refresh” — and doing it during known-quiet windows costs nothing. Not doing it means an OOM at 3am on a Sunday.

Your rolling upgrade pipeline (doc 03) is already the mechanism. Wire in a cron:

kubectl rollout restart deployment/llm-serving-70b --namespace=inference
# every Sunday 03:00 UTC when traffic is at nadir

Blast radius: zero if your rolling upgrade is correct. Failure to do this: silent time-bomb.

---

8. Reading list (verified)

• vLLM production stability RFC series (github.com/vllm-project/vllm/issues, search label:stability)

• Character.AI: “Optimizing AI Inference at Character.AI” — https://blog.character.ai/optimizing-ai-inference-at-character-ai — real numbers on preemption, KV pressure, in-house scheduler tuning

• DoorDash Eng: “How DoorDash Uses ML Model Serving at Scale” — backpressure and admission control lessons

• Anthropic postmortems on the status page (anthropic.com/status) — read the incident timelines

• vLLM --scheduling-policy priority docs

• PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True — PyTorch docs on caching allocator

---

9. Exit test

You have earned this doc when you can, unaided:

1. Given a Prometheus screenshot showing num_preemptions_total rising and gpu_cache_usage_perc steady at 65%, correctly diagnose class-3 fragmentation and prescribe expandable_segments:True plus a weekly restart schedule.

2. Design a per-tenant admission control policy for a Zoho-style multi-tenant CRM-agent workload (three tiers, priority classes, token buckets, LoRA-per-tenant on a shared base) and explain the failure mode each mechanism defends against.

3. Reproduce class-4 (activation-spike) OOM in staging on demand by dispatching a max_model_len - 100 gibberish prompt, then defeat it by tuning chunked-prefill flags — with metrics screenshots proving the fix.

4. Present a one-page “reliability posture” for a hypothetical on-prem 8×H100 Zoho deployment covering the OOM runbook, alert catalogue, weekly refresh cadence, and multi-tenancy isolation strategy — defensible line by line.