04 — Autoscaling LLM Workloads

The scaling reflex you inherited from stateless web services will actively lose money on LLM fleets. GPU utilization is misleading. Memory pressure is misleading. Request rate is misleading. Cold starts are catastrophic. This doc is a rebuild of your autoscaling intuition, from scratch.

1. Why CPU/GPU-util metrics fail for LLMs

Web services autoscaling works because CPU utilization is a proxy for capacity headroom: if CPU is 30%, you have 70% headroom to absorb traffic. That is not true for LLM decode.

Here’s why. Decoding a token on an LLM (batch=1) is memory-bandwidth-bound: the GPU streams every weight through HBM to produce one token. From nvidia-smi’s perspective, the SMs are pegged at ~100% and memory bandwidth is at ~90%. Utilization looks like a fire alarm.

Add a second concurrent request. It gets co-batched. Now you’re computing two tokens per HBM round-trip — same wall-clock, twice the throughput. Utilization? Still 100%. You had massive headroom the util meter didn’t reveal.

At some point, the batch is large enough that the GEMM becomes compute-bound (roofline crossover). Now adding a request does slow the batch. But by then, gpu_util has been reading 100% for a long time.

Conclusion: nvidia_smi_utilization_gpu for an LLM decode workload tells you almost nothing about capacity. If you scale on it, you’ll over-provision by 3–10x. If you don’t scale on it, you’ll under-provision at high real load. It is both wrong at once, in different regimes.

What memory-utilization tells you (and doesn’t)

GPU memory percentage is more meaningful but still misleading. vLLM pre-reserves 90% of VRAM at startup (--gpu-memory-utilization=0.9) whether or not you have any requests. nvidia-smi shows 90% used from the moment the pod is ready. This is a fixed floor, not a signal.

What you actually want is KV cache utilization: how much of the KV block pool is currently allocated to active sequences. vLLM exposes this as vllm:gpu_cache_usage_perc.

2. The metrics that actually predict LLM capacity

Primary autoscaling signals, in order of usefulness:

  1. Queue depth: vllm:num_requests_waiting — the number of requests admitted to the engine but not yet running. This is your “TTFT is degrading” leading indicator. This is the default choice for KEDA-based autoscaling.

  2. KV-cache utilization: vllm:gpu_cache_usage_perc — how full the KV pool is. Approaching 1.0 means the scheduler is about to start preempting/queueing.

  3. Number of running sequences: vllm:num_requests_running — useful as a concurrency signal. Compare against your --max-num-seqs.

  4. TTFT p95 breach rate — you have an SLO like “p95 TTFT < 500ms”. Autoscale when the breach rate is above 5% over the last 60s. This is the SLO-driven approach; more sophisticated.

  5. Preemption count rate: vllm:num_preemptions_total — when the scheduler starts preempting, you are over capacity by definition. Excellent hard-alert signal.

Anti-signals (do NOT scale on these):

  • nvidia_smi_utilization_gpu — misleading (§1).

  • nvidia_smi_memory_used_bytes — pinned by vLLM’s reservation.

  • Container CPU utilization — the CPU is doing very little in a GPU-inference pod; will always look idle.

  • Request rate alone — doesn’t account for prompt length distribution.

3. The KEDA-shaped setup

KEDA (Kubernetes Event-Driven Autoscaler) is the standard mechanism for scaling on custom metrics in 2026. It takes a ScaledObject CRD and drives HPA underneath.

Example: scale on Prometheus queue depth

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: llama-70b-scaler
spec:
  scaleTargetRef:
    name: llama-70b-chat
  minReplicaCount: 2         # never scale to zero for chat (cold start is prohibitive)
  maxReplicaCount: 16
  pollingInterval: 15
  cooldownPeriod: 300        # 5 min after scale-down decision (avoid flapping)
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 300
          policies:
            - type: Percent
              value: 25
              periodSeconds: 60
        scaleUp:
          stabilizationWindowSeconds: 30    # scale up quickly
          policies:
            - type: Percent
              value: 50
              periodSeconds: 30
            - type: Pods
              value: 2
              periodSeconds: 30
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring.svc:9090
        # average queued requests per replica
        query: |
          sum(vllm:num_requests_waiting{app="llama-70b-chat"})
          /
          max(kube_deployment_status_replicas{deployment="llama-70b-chat"})
        threshold: '4'      # start scaling when avg queue depth per replica > 4
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring.svc:9090
        # KV cache pressure as secondary trigger
        query: |
          avg(vllm:gpu_cache_usage_perc{app="llama-70b-chat"})
        threshold: '0.85'

Rules:

  • scaleUp window shorter than scaleDown. Under-provisioning hurts users; over-provisioning wastes money slowly. Prefer to react fast to load and slow to relax.

  • Never scale-to-zero for interactive chat. Cold starts (§5) are 1–10 min; latency-sensitive users won’t tolerate. Save scale-to-zero for batch tiers.

  • minReplicas ≥ 2 for anything with SLOs. Rolling upgrades and node failures need headroom.

4. Workload Variant Autoscaler (WVA) — the llm-d control plane

Mid-2026 development: WVA is the paper-published control plane for llm-d, formally superseding naive HPA for LLM autoscaling. Key ideas:

  • Proactive headroom-based scaling. Rather than reacting to breaches, keep replica headroom sized to serve the p99 forecasted arrival rate for the next N minutes.

  • Fragmentation-aware scale-down. Don’t kill a replica just because aggregate utilization dropped; account for per-replica cache-locality — killing the wrong replica evicts a hot prefix cache and spikes TTFT for the survivors.

  • Multi-model fleet coordination. Free capacity on one model’s pool can absorb another model’s spike, if the pool is heterogeneous-capable.

Reported gains (from WVA paper): ~37% throughput improvement, ~10× fewer SLO failures vs stock HPA on realistic mixed-workload traces.

Status: early adopter territory in mid-2026. If you’re on llm-d, adopt it. If not, its ideas still inform your KEDA configuration — in particular the cache-locality-aware scale-down insight is worth mimicking manually (bias K8s’ scale-down to remove the replica with the lowest recent cache hit rate).

5. Cold-start economics — the reason scale-to-zero mostly doesn’t work

Cold start on an LLM pod, decomposed:

Phase

Time (70B fp16, H100)

What’s happening

K8s admit + schedule

5–30s

scheduler, image pull decision

Container image pull

30–480s

depends on image size + registry proximity

Container start + Python import

5–15s

vLLM import, dependencies

Model weights load

40–180s

140GB @ 1–4 GB/s from disk to GPU

CUDA context + graph capture

10–30s

CUDA graphs for decode shapes

Optional: warm-up requests

5–20s

prime kernels, allocators

Total

95–750s

(1.5 to 12 minutes)

A cold pod that takes 3 minutes to get ready is useless for interactive traffic. If a spike lasts 5 minutes, cold pods contribute for the last 2. You paid for the cold time and the customers left in the first 30 seconds.

Mitigations, ranked by impact

  1. Warm pool with minReplicas ≥ 2. The simplest and most reliable strategy. Cost floor, but predictable latency. Default answer for interactive workloads.

  2. Faster weight load.

    • Local NVMe pre-fetch (see 03_kubernetes_for_gpus.md §8).

    • Parallel S3/MinIO streaming (RunAI Model Streamer, s5cmd, vLLM native S3 support 2026).

    • fp8 or int8 weights halve the load time.

    • Down: 40s → 20s for a 70B on 8×H100.

  3. Slim container image.

    • Multi-stage build. Strip test artifacts, CUDA samples.

    • Base image caching (§7 previous doc).

    • Down: 4 min → 30s.

  4. Model weight snapshotting on shared local NVMe.

    • DaemonSet keeps /opt/models/* warm on every GPU node.

    • New pod skips network transfer entirely.

  5. CRIU + CUDA-Checkpoint (NVIDIA driver 570+).

    • Snapshot the entire GPU process state (weights loaded, CUDA graphs captured, KV pool allocated) to disk.

    • Restore in seconds instead of minutes.

    • Modal reported ~10× cold-start improvement on H100.

    • Community reports: 70B restore in ~2s on H100 with fast NVMe.

    • vLLM RFC #34303 tracks native support. --enable-sleep-mode was broken as of v0.14.0 (issue #32714). Track this closely — it’s the killer feature of 2026–2027.

  6. Serverless GPU providers (Modal, RunPod Serverless, Beam) do the above transparently. For non-latency-critical batch, they can beat DIY.

The economic frame

Ask: “Given my arrival rate, what’s the marginal cost of one warm idle replica vs the p95 latency I gain by keeping it warm?”

Example: an H100 replica at $2.50/hr costs $60/day idle. If keeping it warm saves 3 minutes of degraded p95 latency during 4 daily spikes, worth it. If your traffic is 3 requests/day, obviously not.

Build this calculator into your capacity planning — don’t eyeball it.

6. Heterogeneous fleets and workload-aware routing

Once you have multiple GPU pools (L40S, H100, H200, MI300X) and multiple models (8B utility, 70B chat, 235B MoE), the autoscaling question expands: which pool absorbs the next request?

Routing decisions, at scale

A request router in front of the engine fleet makes these decisions per request:

  1. Model dispatch. Which model does this request want? (Trivial, from the model field.)

  2. Prefix locality. Which replica already holds the KV cache for this request’s prefix? Route there. This is KV-cache-aware routing — 40–80% latency reduction on cache hits.

  3. Load spreading (within a model). If no cache hit, pick the least-loaded replica for that model.

  4. Tier-based routing. Free-tier / batch requests → cheaper pool (L40S). Paid / interactive → premium pool (H100/H200).

Production implementations:

  • SGLang Router (built into SGLang).

  • llm-d Inference Gateway (Kubernetes Gateway API Inference Extension).

  • NVIDIA Dynamo router.

  • Custom Envoy filter reading vllm:gpu_prefix_cache_hit_rate metric.

The pool-shaped autoscaler

Each pool scales independently, but the router influences the effective demand:

  • Traffic on Pool A grows → autoscaler adds A replicas.

  • Router also starts spilling A’s overflow to compatible Pool B (e.g., H100 → H200).

  • Pool B absorbs some traffic before its own autoscaler reacts.

This is exactly what WVA (§4) formalizes.

7. Tiered SLO design (batch vs interactive)

Batch tier

  • Priority: low.

  • SLO: throughput, not latency. “Complete the batch by 6am.”

  • Runs on: L40S, MI300X, spot instances.

  • Scaling: aggressive scale-to-zero at low queue; scale-up on batch admission.

  • vLLM config: high --max-num-seqs, high --max-num-batched-tokens, no --enable-chunked-prefill (prefill throughput matters most).

Interactive tier

  • Priority: high.

  • SLO: p95 TTFT < 500ms, p95 ITL < 50ms.

  • Runs on: H100/H200.

  • Scaling: warm pool minReplicas ≥ 2; fast-react scaleUp window.

  • vLLM config: moderate --max-num-seqs (bounded batching to keep ITL down), chunked prefill on.

Do not put both on the same pool. They fight for capacity in ways that break both SLOs.

8. What Zoho actually gets from this

Zoho on-prem customer: typically 8–32 GPUs total, mixed vintage. Homogeneous scaling policies will waste money. Design a tiered platform:

  • Interactive tier on H100/H200 (or MI300X) with warm pool.

  • Batch tier on L40S or older A100s.

  • KEDA autoscaling on vllm:num_requests_waiting per pool.

  • Prefix-cache-aware routing (huge win for CRM copilot loops, which reload system prompts every turn).

Zoho SaaS side: you probably have burstier and cheaper hardware (spot). Scale-to-zero on batch tier saves real money. Warm pool on chat. Different game entirely.

Building this playbook — documented, benchmarked, defensible — and presenting it as “how Zoho does LLM autoscaling” is a staff-engineer-level artifact. 14_projects.md includes it as a named deliverable.

Reading list

  1. KEDA docs, Prometheus scaler guide.

  2. WVA arxiv paper (2603.09730) — read the fragmentation-aware scale-down section carefully.

  3. Spheron blog: “KEDA + Knative GPU autoscaling & LLM cold starts.”

  4. vLLM metrics reference: docs.vllm.ai/en/stable/design/metrics.

  5. Character.AI “Optimizing AI inference” blog series — look at how they justify their capacity choices.

Exit test

  1. On a whiteboard, list the top 5 autoscaling signals for LLMs and the top 3 anti-signals.

  2. Design a KEDA ScaledObject for a 70B chat model with p95 TTFT SLO of 400ms.

  3. Given a workload with 3 spikes/day, each 90s long, decide: scale-to-zero + fast-warm or warm pool? Show the math.

  4. Explain to a K8s engineer why nvidia_smi_utilization_gpu is the wrong autoscaling metric for LLM decode. Under 2 minutes.