03 — Kubernetes for GPUs¶

Kubernetes was designed for stateless request/response services with 100MB container images. LLM serving is stateful (KV cache), latency-sensitive (streaming), heavyweight (200GB model images), and hardware-specialized (GPU affinity, NVLink topology). Making K8s serve LLMs well requires unlearning some K8s reflexes.

Why K8s at all¶

Because your enterprise customer runs it. Because your ops team knows it. Because the CNCF ecosystem (Prometheus, OTel, KServe, llm-d, KEDA) speaks it natively. And because the alternative — bespoke bare-metal orchestration — is where operational disasters live at 3am.

The cost is that K8s is not built for LLMs. This doc is the list of adaptations that make it work.

1. The NVIDIA GPU stack on Kubernetes¶

At minimum you need three things:

NVIDIA driver on the node (Linux, matched to CUDA runtime version).
NVIDIA container toolkit (formerly nvidia-docker) so containers can see GPUs.
NVIDIA device plugin for Kubernetes — a DaemonSet that advertises nvidia.com/gpu as a schedulable resource.

Rather than installing these piecemeal, use the NVIDIA GPU Operator (helm install gpu-operator nvidia/gpu-operator -n gpu-operator --create-namespace). It:

Installs drivers if you want (or defers to host drivers).
Deploys the device plugin.
Deploys DCGM Exporter for GPU-level Prometheus metrics.
Deploys the MIG manager (see §3).
Handles GPU Feature Discovery (labels nodes with product name, memory, compute cap).
Manages node-level GPU driver upgrades (with careful drain).

The GPU Operator is table stakes. You are not building a serious K8s LLM platform without it in 2026.

Requesting a GPU (the basic case)¶

resources:
  limits:
    nvidia.com/gpu: 1

What this actually does: the device plugin injects the appropriate device files (/dev/nvidia*) and env vars (NVIDIA_VISIBLE_DEVICES) into your container. limits.nvidia.com/gpu must equal requests.nvidia.com/gpu — K8s treats GPUs as “extended resources” which must be equal.

Node labels & taints¶

Standard pattern: label GPU nodes and taint them so only GPU workloads land there.

# node label
nvidia.com/gpu.product: NVIDIA-H100-SXM5-80GB
nvidia.com/gpu.memory: 81559
nvidia.com/gpu.count: 8

# taint
key: nvidia.com/gpu
effect: NoSchedule

Pods tolerate the taint + nodeSelector on the product label. This prevents rando CPU workloads from crowding your (expensive) GPU nodes.

2. Node pools & GPU heterogeneity¶

A mature LLM fleet is heterogeneous by design. You do NOT want a single H100 pool.

Node pool	GPUs	Purpose
`pool-h100`	8×H100 SXM	70B chat, high QPS
`pool-h200`	8×H200 SXM	100B+ chat, long context
`pool-l40s`	4×L40S	small models (8B), batch summarization
`pool-cpu`	CPU-only	tokenization microservices, embeddings via CPU

Advantages:

Right-size to workload. 8B on H100 wastes ~90% of the memory-bandwidth capacity.
Independent autoscaling. Small-model traffic doesn’t force large-model provisioning.
Vendor mix. Some pools MI300X, some H100 — hedge against supply crunches (real in 2026).

Use nodeSelector or the newer nodeAffinity with preferredDuringSchedulingIgnoredDuringExecution to steer workloads.

The Zoho on-prem angle¶

On-prem customers rarely have homogeneous fleets. They have “whatever we could procure this quarter.” Your job is to make the platform absorb that heterogeneity gracefully. Cost model per pool (see 09_cost_modeling.md), route the workload to the pool whose $/token is lowest for that workload’s shape.

3. GPU sharing: MIG vs time-slicing vs MPS¶

When a workload doesn’t need a full GPU, you have three options. They are NOT interchangeable.

3.1 Time-slicing (software multiplexing)¶

Enabled via NVIDIA device plugin config: nvidia.com/gpu.sharing-strategy=time-slicing, nvidia.com/gpu.replicas=4.
Advertises 1 physical GPU as 4 virtual GPUs.
Pods share the GPU sequentially — the CUDA scheduler round-robins.
No memory isolation. Total pod VRAM must fit within the physical GPU.
No fault isolation. One pod’s OOM/hang can affect all sharers.
When to use: dev environments, low-traffic model serving, workloads with bursty and non-overlapping demand.
When NOT to use: multi-tenant customer isolation, latency-sensitive prod, anything where a noisy neighbor is unacceptable.

3.2 MIG — Multi-Instance GPU (hardware partitioning)¶

Available on: A100, A30, H100, H200. Not on L40S, RTX Pro 6000, MI300X, or consumer cards.
Hardware-partitions the GPU into isolated instances with dedicated SMs and dedicated VRAM slices.
Example H100 profiles: 1g.10gb, 2g.20gb, 3g.40gb, 4g.40gb, 7g.80gb. The digits are (compute slice count).(memory GB).
Actual isolation — memory, fault, and performance.
Configured via node label: nvidia.com/mig.config=all-1g.10gb (all-uniform) or mixed (per-GPU profiles).
Requires reboot / GPU reset to change profile. Static planning.
Pods request e.g. nvidia.com/mig-1g.10gb: 1.
When to use: multi-tenant serving where isolation matters; running many small models on a big GPU; QoS tiers.
When NOT to use: you need >7 concurrent workloads per GPU (MIG maxes at 7 slices on H100); you need to change profiles frequently.

3.3 MPS — Multi-Process Service¶

CUDA-level concurrency: multiple processes submit kernels to one MPS daemon, sharing the SMs simultaneously.
Best throughput for cooperative workloads.
Zero fault isolation — one process crash can kill peers.
Use only for trusted co-scheduled workloads (e.g., one team’s models). Never expose across security boundaries.

3.4 DRA — Dynamic Resource Allocation¶

K8s beta feature (stable in 1.34+) enabling more flexible GPU allocation than the extended-resource model. In 2026 it’s the emerging standard for expressing “give me a GPU with ≥80GB HBM and NVLink to another GPU in the same request.” Watch this space; not yet the default in production but is where the ecosystem is heading.

3.5 The decision matrix¶

Scenario	Pick
Dev cluster, share GPUs across engineers	Time-slicing
Prod, multiple tenants, isolation required	MIG
Prod, one tenant, want max throughput on small model	MPS (trusted) or MIG
Prod on L40S/RTX (no MIG hardware)	Time-slicing or full-GPU pods
“We want to run 20 fine-tunes on 1 H100”	vLLM `--enable-lora`, not GPU sharing

That last row is critical: for many-fine-tunes-per-model, use vLLM’s LoRA hot-swap, not GPU partitioning. GPU sharing splits the GPU; LoRA sharing splits the compute inside a single vLLM process with shared base weights. LoRA is dramatically more efficient.

4. Topology-aware scheduling¶

On 8-GPU nodes, GPUs are wired with a topology:

Full NVLink mesh (8 GPUs directly connected via NVSwitch on HGX H100/H200/B200 boards).
Or partial (PCIe switches between GPU pairs).

A TP=4 workload wants all 4 GPUs on the same NVLink domain — otherwise TP all-reduces go over PCIe/RoCE and throughput collapses.

Solutions in 2026:

NRI (Node Resource Interface) plugins. Kubelet-level hooks that let a topology-aware scheduler pin containers to specific GPU indices with NVLink locality.
NVIDIA’s device plugin --pass-device-specs + --gpu-selection-policy=best-effort — the plugin now understands NVLink topology when picking which GPUs to expose to a container.
Topology Manager (kubelet built-in) — aligns GPU, CPU, and memory NUMA nodes. Enable --topology-manager-policy=single-numa-node.

Verification: run nvidia-smi topo -m inside the pod and check that the GPUs share NV12 (NVLink) rather than PIX (PCIe) or SYS (across sockets).

When it matters: any TP > 1. If you skip topology-aware scheduling on a TP=8 70B workload, you can lose 30–50% of your throughput to bad interconnect placement.

5. Liveness, readiness, and the model-load-≠-ready gotcha¶

THIS IS THE #1 K8S-LLM PRODUCTION BUG. If you take one thing from this doc, take this.

The naive setup that breaks¶

readinessProbe:
  httpGet:
    path: /health
    port: 8000
  initialDelaySeconds: 30
  periodSeconds: 10

vLLM starts the HTTP server almost immediately — within seconds. But loading a 70B model into GPU memory takes 60–300 seconds. During that window, /health returns 200 while /v1/chat/completions returns 503 or hangs.

K8s marks the pod ready. The Service starts sending it traffic. Every request fails. Repeatedly. The pod gets marked “crashed” via liveness or its request timeout, K8s restarts it, model loads again, same failure.

Result: rolling deployment brownout for the entire duration of the model load, times the number of replicas.

The correct setup¶

Use vLLM’s actual readiness endpoint that reports model-loaded state:

# vLLM 0.6+ exposes /health only when model is loaded and engine is ready
readinessProbe:
  httpGet:
    path: /health
    port: 8000
  # First check: enough time for model to load. 70B on H100 SXM ≈ 90-120s.
  initialDelaySeconds: 120
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 6

livenessProbe:
  httpGet:
    path: /health
    port: 8000
  # Liveness must NOT trip during startup. Use a startupProbe instead.
  periodSeconds: 30
  timeoutSeconds: 5
  failureThreshold: 4

startupProbe:
  httpGet:
    path: /health
    port: 8000
  # Model load. Fail after ~10 min for a big model.
  periodSeconds: 15
  failureThreshold: 40

Rules:

Use startupProbe for model load. Livenessprobe only kicks in after startup succeeds. Otherwise a slow model load will be interpreted as a liveness failure and K8s will restart-loop the pod.
Readiness reflects engine-ready, not process-alive. vLLM’s /health does the right thing in recent versions — verify per version.
initialDelaySeconds sized to p95 model load time, not average. Long tail is real (page cache misses, slow NFS).

For custom servers¶

If you serve behind a custom Python wrapper, expose two endpoints:

/livez — process is alive (returns 200 always).
/readyz — model loaded, GPU allocated, warm-up completed (200 only when true).

6. Graceful drain¶

When a pod is deleted (rolling upgrade, scale down, node maintenance), you must:

Stop accepting new requests (readiness = 0 → Service stops routing).
Let in-flight streaming requests finish — or send them a graceful finish_reason: length shutdown message.
Then terminate.

K8s mechanics:

Pod deletion sends SIGTERM, then waits terminationGracePeriodSeconds (default 30, way too short for LLMs), then SIGKILL.
Set terminationGracePeriodSeconds: 600 (10 min) for chat models. Longer for models producing long outputs.
Handle SIGTERM in your wrapper: mark unready, close listener, drain, exit.
Use a preStop lifecycle hook to sleep briefly so kube-proxy’s iptables rules update before the process starts refusing connections.

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 15 && curl -X POST localhost:8000/shutdown"]
terminationGracePeriodSeconds: 600

vLLM 2026 has a --enable-shutdown-endpoint flag that enables POST /shutdown to drain gracefully. Verify per version.

7. Rolling upgrades of 200GB model images¶

Enterprise reality: your Llama-70B image is 140GB (weights + framework + CUDA libs). Rolling this to 20 replicas is a 2.8TB registry read.

The naive path¶

Every pod pulls the full image from your registry.
Registry saturates. Pull times blow past imagePullDeadlineSeconds.
Rolling upgrade drags for 20–60 minutes.

The fixes¶

Split the image.
- Base container image (~5–10GB): OS, CUDA, PyTorch, vLLM. Rarely changes.
- Model weights: NOT in the image. Mount as a volume (see below).
Store weights on a shared NVMe or object store, not in the image.
- Local NVMe cache per node. DaemonSet pre-fetches weights to /opt/models. Pod mounts as hostPath.
- Read-only PVC backed by fast shared storage (NFS-over-NVMe, Ceph, MinIO with proxy caching).
- S3-compatible + fsspec streaming — vLLM can now (2026) stream weights from S3/MinIO directly at startup, in parallel.
Use a registry pull-through cache. Harbor, Zot, or Docker Distribution in mirror mode. Deploy in the same DC as GPU nodes. First pull hits upstream once; subsequent pulls are LAN speed.
Pre-pull on nodes. DaemonSet that runs crictl pull on new image tags before the Deployment rolls. Now the Deployment’s rolling update is instant — image already on every node.
Rolling strategy for LLMs is NOT RollingUpdate with default maxSurge/maxUnavailable. Use:
```
strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 1        # bring up one extra at a time (needs GPU headroom)
    maxUnavailable: 0  # never drop below N ready
```
Combine with a canary via traffic splitting (Istio, KServe, or Argo Rollouts) — route 5% of traffic to the new revision, watch SLO, ramp.

The Zoho on-prem gotcha¶

At customer sites you often DON’T control the registry. Air-gap deployments (see 07_on_prem_enterprise.md) require:

Registry mirror deployed inside the air-gap.
Signed & verified images.
Provenance chain (SBOM — Software Bill of Materials).
Model weights on internal object storage with hash verification on load.

Design decision: for on-prem, always ship model weights separately from container image. Customer’s security team needs to sign the weights independently.

8. Storage patterns for model weights¶

Options, ordered by increasing cost and speed:

Option	First-pod load time (70B fp16, 140GB)	Cold-node load
Pull from S3 at startup, sequential	300–600s	300–600s
Pull from S3, parallel streaming (vLLM 2026)	90–180s	90–180s
Shared NFS/CephFS	120–300s	120–300s
PVC on local SSD, ReadWriteMany	60–180s	full pull first
Local NVMe hostPath (pre-fetched)	40–80s	one-time pre-fetch
tmpfs (RAM-backed) after first load	40–80s	RAM cost

Recommended default for on-prem H100/H200: local NVMe hostPath with a DaemonSet pre-fetcher. NVMe reads at 5–10 GB/s parallel; 140GB in ~30s.

For cloud: parallel S3 streaming via RUN_AI_MODEL_STREAMER or s5cmd or vLLM native streaming.

9. Networking¶

For multi-node inference (TP across nodes) you need:

RDMA (InfiniBand or RoCE v2). Not TCP. NCCL over IB is 5–10x lower latency than over Ethernet TCP.
SR-IOV or CNI plugins that pass the RDMA interface through to pods.
NCCL topology detection — set NCCL_SOCKET_IFNAME and verify with NCCL_DEBUG=INFO on first run.

On-prem enterprise customers often don’t have IB fabric. Serving a model that requires cross-node TP on such customers is an anti-pattern; keep tensor parallelism within a single node’s NVLink domain.

10. Reference Deployment skeleton¶

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llama-70b-chat
  labels: { model: llama-70b, tenant: shared }
spec:
  replicas: 4
  strategy:
    type: RollingUpdate
    rollingUpdate: { maxSurge: 1, maxUnavailable: 0 }
  selector: { matchLabels: { app: llama-70b-chat } }
  template:
    metadata:
      labels: { app: llama-70b-chat }
    spec:
      terminationGracePeriodSeconds: 600
      nodeSelector: { nvidia.com/gpu.product: NVIDIA-H100-SXM5-80GB }
      tolerations: [{ key: nvidia.com/gpu, effect: NoSchedule }]
      containers:
      - name: vllm
        image: registry.zoho.internal/inference/vllm:0.7.3-cuda12.4
        args:
          - "vllm"
          - "serve"
          - "/models/llama-3.3-70b-instruct"
          - "--tensor-parallel-size=4"
          - "--max-model-len=8192"
          - "--max-num-seqs=128"
          - "--gpu-memory-utilization=0.92"
          - "--kv-cache-dtype=fp8_e4m3"
          - "--served-model-name=zoho-llama-70b"
        ports: [{ containerPort: 8000 }]
        resources:
          limits: { nvidia.com/gpu: 4 }
          requests: { nvidia.com/gpu: 4, memory: 200Gi }
        volumeMounts:
          - { name: models, mountPath: /models, readOnly: true }
          - { name: shm, mountPath: /dev/shm }
        startupProbe:
          httpGet: { path: /health, port: 8000 }
          periodSeconds: 15
          failureThreshold: 40
        readinessProbe:
          httpGet: { path: /health, port: 8000 }
          periodSeconds: 10
          failureThreshold: 3
        livenessProbe:
          httpGet: { path: /health, port: 8000 }
          periodSeconds: 30
          failureThreshold: 4
        lifecycle:
          preStop:
            exec: { command: ["/bin/sh", "-c", "sleep 15"] }
      volumes:
        - name: models
          hostPath: { path: /opt/models, type: Directory }
        - name: shm
          emptyDir: { medium: Memory, sizeLimit: 32Gi }

This is the boring, correct skeleton. Copy it, adapt it, don’t re-invent it.

Reading list¶

NVIDIA GPU Operator docs — docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/
MIG User Guide — docs.nvidia.com/datacenter/tesla/mig-user-guide/
KServe LLM Runtime docs.
Kubernetes DRA docs (K8s 1.34+).
Red Hat’s OpenShift AI documentation for a mature enterprise stack.

Exit test for this doc¶

Draw the layers: node → device plugin → GPU Operator → workload. Label every component.
Given “customer wants to serve 3 tenants isolated on 1×H100,” recommend MIG configuration + K8s manifest.
Explain the model-load-≠-ready bug to a K8s engineer who has never served an LLM. Under 3 minutes.
Design the model-weight distribution strategy for a 5-node on-prem cluster serving Llama-70B, with a defensible answer for cold-restart time.