01 — Engines as Products

By the end of Phase 4 you knew how vLLM and SGLang worked. In Phase 7 you learn to configure them like products, and to understand the orchestration layer that sits above them. The distinction matters: a running engine is a research artifact; a configured, orchestrated, observable engine with SLOs is a product.

The taxonomy that keeps you sane

There are three layers here, and confusing them is the #1 mistake made in engine selection meetings:

Layer

What it does

Examples

Engine

Scheduler + memory manager + kernels. Runs models.

vLLM, SGLang, TensorRT-LLM, llama.cpp

Model server

Wraps engine(s) with HTTP/gRPC, model management, multi-model routing.

NVIDIA Dynamo-Triton (formerly Triton)

Orchestrator

Fleet-level: routing, autoscaling, cache-aware LB, P/D disaggregation, K8s integration.

NVIDIA Dynamo, llm-d, KServe, Ray Serve

You can run an engine alone (fine for a single node). You can run an engine behind a model server (adds standardized management). You need an orchestrator when you have >1 replica and care about cache locality, autoscaling, or disaggregated serving.

1. vLLM as a product — the config surface that matters

vLLM has ~150 CLI flags. Most don’t matter. These do, grouped by what they buy you:

Throughput / batching

  • --max-num-seqs (default 256) — max concurrent sequences per iteration. Raise this to increase batch size for throughput; lower it if you’re memory-pressured. Interacts with GPU memory: max_seqs × avg_kv_per_seq ≤ available KV pool.

  • --max-num-batched-tokens — token budget per iteration. This is the chunked prefill knob. Default is usually 2048–8192. Lower = smoother ITL (prefills don’t hog steps); higher = better prefill throughput.

  • --max-model-len — hard context cap. Setting this lower than the model’s native context (e.g., 8k when the model supports 128k) frees KV memory and raises achievable batch size dramatically. Always set this to the shortest length your workload actually needs.

Memory / KV cache

  • --gpu-memory-utilization (default 0.90) — fraction of GPU memory vLLM reserves at startup. Push to 0.94 for single-tenant nodes; drop to 0.85 if you have other workloads or need OOM headroom for long prefills.

  • --kv-cache-dtype — auto / fp8 / fp8_e5m2 / fp8_e4m3. fp8 KV halves KV memory for negligible quality loss on most workloads; on Hopper/Ada, use fp8_e4m3. On ROCm/MI300X, fp8_e4m3 is supported.

  • --block-size — KV block size in tokens (8/16/32/64/128; CUDA cap is 32). Default 16 is fine; 32 helps very-long-context throughput at the cost of fragmentation.

  • --swap-space (GB) — CPU swap for KV preemption. Not free (PCIe round-trip); prefer recompute unless prompts are truly expensive to re-prefill.

Latency / caching

  • --enable-prefix-caching — default ON in recent versions. For agentic workloads, this is the single most impactful flag on the whole CLI. Verify hit rate via the metric vllm:gpu_prefix_cache_hit_rate (see 05_observability.md).

  • --enable-chunked-prefill — default ON in recent versions. Splits long prefills into chunks co-batched with decode. Massively improves ITL for mixed traffic. Disable only if you have prefill-only workloads.

  • --long-prefill-token-threshold — controls what counts as “long” for chunking. Defaults to 4% of max context.

Speculative decoding

  • --speculative-model, --num-speculative-tokens, --speculative-max-model-len. Set a small drafter (e.g., a 1B Llama or an EAGLE head) to speed decode. Verify acceptance rate via vllm:spec_decode_num_accepted_tokens_total; if <60%, spec is costing you.

  • Known 2026 landmine: vLLM issue #43559 (May 2026) — prefix-caching + MTP spec-decoding on Qwen3.6 causes a 20% accuracy drop. Never enable spec-decode in prod without a quality gate.

Execution

  • --enforce-eager — disables CUDA graph capture. Only use for debugging. Eager mode adds ~10–30% decode latency at low batch.

  • --max-seq-len-to-capture (formerly max-context-len-to-capture) — biggest shape CUDA graphs cover. Requests longer than this fall back to eager. Set to your p99 sequence length.

  • --quantization — fp8 / awq / gptq / gptq_marlin / awq_marlin / compressed-tensors / bitsandbytes / hqq / moe_wna16 / etc. Marlin variants are what you actually want on Ampere/Hopper for 4-bit.

LoRA multi-tenancy

  • --enable-lora, --max-loras, --max-lora-rank, --max-cpu-loras. Serving many fine-tunes off one base model. This is how you host 200 customer-specific fine-tunes economically. (See 07_on_prem_enterprise.md for why this matters for Zoho verticals.)

Metrics / API

  • /metrics endpoint (Prometheus) is on by default when the OpenAI-compatible server is running. See 05_observability.md for the full metric list.

A minimal production vllm serve invocation (annotated)

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 4 \                    # 4× GPU per replica (fits fp16 70B on 4×H100 with room)
  --max-model-len 8192 \                        # cap context; frees KV
  --max-num-seqs 128 \                          # concurrent sequences
  --max-num-batched-tokens 4096 \               # chunked prefill token budget
  --gpu-memory-utilization 0.92 \
  --kv-cache-dtype fp8_e4m3 \                   # halves KV, negligible quality hit
  --enable-prefix-caching \                     # default on, name explicit for review-ability
  --enable-chunked-prefill \                    # default on, ditto
  --max-seq-len-to-capture 8192 \               # CUDA graphs cover the full context
  --served-model-name zoho-llama-70b \          # what OpenAI clients see
  --port 8000

Every flag above is a defensible line — you can justify each with arithmetic (KV memory, latency budget) or a benchmark. If you can’t defend one, remove it.

Verification source: vLLM engine args reference at https://docs.vllm.ai/en/stable/serving/engine_args.html. Flag names change between minor versions; always cross-check the /version endpoint against the docs page.

2. SGLang production knobs

SGLang is vLLM’s closest peer and often better for agentic workloads because RadixAttention is more aggressive than vLLM’s prefix caching:

  • --tp / --dp — tensor and data parallel.

  • --mem-fraction-static — analogue of gpu-memory-utilization; typical 0.85–0.90.

  • --enable-radix-cache — the reason to use SGLang. DeepSeek-in-production reports 342B of 608B input tokens (56%) hit the on-disk RadixAttention cache. Agents with fixed system prompts + tool definitions see 75–95% hit rates.

  • --max-running-requests — analogue of max-num-seqs.

  • --chunked-prefill-size — chunked prefill token budget.

  • --enable-torch-compile — Turns on torch.compile for decoding, which composes well with SGLang’s kernels but has warm-up cost.

  • --attention-backend — flashinfer (default; usually fastest) / triton / torch-native.

  • --kv-cache-dtype fp8_e5m2 — same trick as vLLM.

  • --enable-metrics — exposes Prometheus.

  • SGL Router (separate process) — cache-aware load balancer; hash requests to replicas holding matching prefixes. This is the “cache-aware routing” feature you’d otherwise pay for via llm-d.

When to pick SGLang over vLLM:

  • Heavy structured-output workloads (XGrammar is native).

  • Multi-turn agent workloads where prefix hit rate is the whole game.

  • You want native cache-aware routing without adopting llm-d.

When to pick vLLM over SGLang:

  • Sheer breadth of model support (vLLM lands new architectures faster).

  • Larger community, more production battle-tested for chat use cases.

  • Better tooling around LoRA multi-tenancy.

Truth: for most Zoho workloads, either works. The choice is 60% team familiarity, 40% specific-workload benchmarks. Run both against your actual traffic before committing.

3. TensorRT-LLM — the compiled-engine world

TensorRT-LLM is architecturally different: instead of a general Python scheduler over PyTorch, you compile a model into a serialized engine against a specific GPU + shape range + precision, and serve that binary blob.

  • Builder flow: trtllm-build --checkpoint_dir ... --output_dir ... --max_batch_size N --max_input_len I --max_seq_len S --gemm_plugin fp16 --strongly_typed. The compile can take 10–60 minutes.

  • Peak numbers: on the same H100, TensorRT-LLM often beats vLLM by 10–30% on well-matched shapes. On mismatched shapes it can lose badly because the compile is rigid.

  • The rigidity tax: the compiled engine is tied to a GPU family, precision, and shape envelope. If you change any of these you rebuild. In a fleet of mixed H100/H200 or with heterogeneous prompt lengths, this is expensive.

  • When it justifies itself: single-model, single-GPU-SKU, latency-sensitive, throughput-critical deployments — e.g., a customer-specific 70B chat model on dedicated 8×H100 that you’re going to leave alone for 6 months. The 15% throughput gain pays for the ops complexity.

  • When it doesn’t: anything where model or hardware is changing weekly. You’ll spend more time rebuilding engines than serving traffic.

Status 2026: actively developed by NVIDIA, primary path for Dynamo Blackwell perf claims, tighter integration with Dynamo than with anything else.

4. NVIDIA Dynamo — the orchestration story, verified

Verified status (as of mid-2026):

  • Dynamo 1.0 launched at GTC March 16 2026. Open source (Apache 2.0), Python + Rust. GitHub: github.com/ai-dynamo/dynamo.

  • NVIDIA’s positioning: “inference operating system for AI factories.” Concrete: it’s a distributed serving orchestrator built for LLM/generative-AI-scale inference, with disaggregated prefill/decode, KV-cache-aware routing, KV-cache offload to storage (via NIXL), and SLO-driven autoscaling.

  • Backends supported: PyTorch, vLLM, SGLang, TensorRT-LLM. So Dynamo does not replace vLLM — it orchestrates it.

  • On Kubernetes: ships with Grove (K8s operator).

Dynamo vs Triton — the naming confusion resolved

  • Triton Inference Server was renamed to NVIDIA Dynamo-Triton on March 18 2025.

  • Dynamo-Triton = the general model server (TensorRT/PyTorch/ONNX/OpenVINO/Python/RAPIDS backends). It’s still shipped, still supported. Best for mixed-modality fleets (LLM + vision + tabular ML on the same platform).

  • Dynamo (the new thing) = LLM-specialized orchestrator with disaggregated serving, prefix-cache-aware routing, KV-cache offload. It sits above engines like vLLM/SGLang/TensorRT-LLM.

  • If someone says “should we use Triton or Dynamo?” the correct answer is: “Depends. Dynamo-Triton for general model serving; Dynamo for LLM-scale inference orchestration; they’re both NVIDIA-supported and can coexist.”

When Dynamo actually pays off

  • You have >8 GPUs and >1 model, and want cache-aware routing.

  • You want disaggregated prefill/decode (see Phase 6 §6.2 — DistServe/Mooncake) and don’t want to build it yourself.

  • You’ve committed to NVIDIA hardware (Dynamo is not vendor-neutral).

When it doesn’t

  • You have one 8-GPU node. Just use vLLM with --tensor-parallel-size 8. Dynamo adds ops complexity that pays off at fleet scale, not single-node.

  • You want CNCF-governed, K8s-first, vendor-neutral. Use llm-d instead.

Verification source: developer.nvidia.com/dynamo, developer.nvidia.com/dynamo-triton, GTC 2026 announcements.

5. llm-d — verified and important

Verified status (mid-2026):

  • CNCF Sandbox project since March 24 2026 (announced KubeCon EU 2026 Amsterdam).

  • Founding companies: Red Hat, Google Cloud, IBM Research, CoreWeave, NVIDIA. (Yes — NVIDIA co-founds a project that competes with its own Dynamo. Read: Dynamo is where NVIDIA extracts value on its hardware; llm-d is the vendor-neutral K8s story that keeps hyperscalers and Red Hat happy. Both are real.)

  • Architecture: Kubernetes-native distributed inference framework built on top of vLLM. Uses Kubernetes Gateway API Inference Extension for cache-aware routing. Uses vLLM’s KV Connector API for prefill/decode disaggregation, with NIXL as the KV transport.

  • The pitch: “Dynamo for people who live on Kubernetes.”

llm-d vs Dynamo — the pragmatic decision

Dimension

Dynamo

llm-d

Governance

NVIDIA-owned OSS

CNCF Sandbox

Primary substrate

Bare metal, DGX, K8s (via Grove)

Kubernetes-first

Engine

vLLM / SGLang / TRT-LLM / PyTorch

vLLM (canonical)

Vendor lock-in

NVIDIA-favored (TRT-LLM path is best)

Neutral

Maturity mid-2026

Newer, moves fast

Newer, moves fast

Zoho fit

If on-prem H100 fleet and NVIDIA-blessed reference architectures matter

If K8s-native stack and CNCF governance matter to enterprise buyers

For Zoho specifically: enterprise customers with existing K8s/OpenShift standardization will ask about llm-d. Being able to say “yes, our stack is CNCF-governed and K8s-native” is a security-review shortcut worth investing in. Dynamo is a better answer if the customer is buying an NVIDIA reference architecture.

Verification source: github.com/llm-d/llm-d, llm-d.ai/blog/llm-d-announce, KubeCon EU 2026 recordings.

6. Triton Inference Server (now Dynamo-Triton) — the general workhorse

If you have a mixed workload — LLMs, YOLO for object detection, an XGBoost model, an ONNX classifier — Dynamo-Triton is still the sensible model server. It gives you:

  • Model repository with versioning.

  • Ensemble models (pipeline multiple models with one HTTP call).

  • Backends for TRT/PyTorch/ONNX/OpenVINO/Python/RAPIDS.

  • gRPC + HTTP + streaming, standardized metrics.

  • Model warmup, priority queues, dynamic batching (for non-LLM models — LLMs should be behind vLLM/SGLang not Triton’s naive batcher).

The gotcha: don’t serve LLMs directly out of Triton’s Python backend. Wrap vLLM/TRT-LLM as a Triton backend, or (better in 2026) use Dynamo-the-orchestrator with Dynamo-Triton as one of the engines in the fleet.

7. KServe — Kubernetes-native model serving

  • CNCF Incubating project. Standard CRD is InferenceService (v1); for LLMs, LLMInferenceService is the newer CRD (2026).

  • What it gives you: declarative model deployment (kubectl apply -f my-llm.yaml), autoscaling (via KEDA or its own knative-based autoscaler), traffic splitting for canary/A-B, model explainability hooks, standardized metrics.

  • How it composes with llm-d and vLLM: KServe manages the lifecycle (deploy, upgrade, roll back, autoscale). llm-d provides the routing intelligence (KV-cache-aware, P/D disaggregation) above multiple KServe-managed inference services. vLLM is the engine underneath.

  • Verification source: developers.redhat.com/articles/2026/04/21/kserve-llm-d-optimized-gen-ai-inference.

When KServe is the right hammer

  • You’re on Kubernetes (or OpenShift, especially in enterprise).

  • You want GitOps for model deployments (ArgoCD/Flux + KServe CRDs).

  • You want traffic-splitting for safe rollouts of new models or engine versions.

When it isn’t

  • You have a single node and don’t need K8s at all.

  • Your ops team can’t own another K8s operator.

8. Ray Serve — the Python-native alternative

  • Model: actors + deployment graphs. Instead of “one model per service” you compose multiple models into a Python-native pipeline.

  • Where it shines: heavy request-shaping logic — an inference request that hits a small router model, then a retrieval step, then the main LLM, then a re-ranker, all in one graph. This maps very directly onto agent-shaped workloads.

  • LLM specifics 2026: Ray Serve has first-class vLLM/SGLang integration. serve.llm module. Autoscaling on request queue length. Fractional GPU allocation (num_gpus=0.5).

  • Trade-off: Ray is another distributed system to operate. If you already run Kubernetes and one of your senior engineers has scars from a bad Ray cluster upgrade, KServe is safer.

Zoho angle: Ray Serve is particularly good if you’re prototyping agentic inference pipelines that call multiple models per request. Your day job is close enough to this that Ray Serve is worth 3 days of hands-on time, at minimum.

The decision tree

At risk of over-simplifying:

Single-node, one model, on-prem?             → vLLM alone (behind nginx if you must)
Single node, mixed model types?               → Dynamo-Triton
Multi-node LLM fleet, NVIDIA reference?       → NVIDIA Dynamo above vLLM/TRT-LLM
Multi-node LLM fleet, Kubernetes-first?       → llm-d above vLLM (+ KServe for lifecycle)
K8s + GitOps + traffic splitting matters?     → KServe (+ llm-d if cache-aware routing needed)
Complex request-shaping / model composition?  → Ray Serve
Squeezing the last 15% on fixed model+HW?     → TensorRT-LLM under Dynamo

The reading list (do these this month)

  1. vLLM engine args docs — read the whole page. docs.vllm.ai/en/stable/serving/engine_args.html

  2. SGLang production guide — docs.sglang.ai/backend/server_arguments.html

  3. NVIDIA Dynamo blog + docs — start at developer.nvidia.com/dynamo

  4. llm-d architecture doc + KubeCon EU 2026 talks

  5. KServe LLMInferenceService announcement (Red Hat Developers, April 2026)

  6. Ray Serve LLM docs

Exit test for this doc

You can, unaided, at a whiteboard:

  1. Draw the three-layer taxonomy (engine / model server / orchestrator) and place vLLM, SGLang, TRT-LLM, Dynamo-Triton, Dynamo, llm-d, KServe, Ray Serve in the correct layers.

  2. Given a scenario (“4-node H200 cluster serving 3 different fine-tunes of Llama 3.3 70B for 3 customer tenants, must support prefix-cache-aware routing, K8s-native, quarterly model upgrades”), select the stack and defend every choice.

  3. Recite the top 12 vLLM flags from memory, with their default and effect.

  4. Explain the practical difference between Dynamo and Dynamo-Triton to a technically-literate but not LLM-native architect in under 90 seconds.

If any of these are shaky, the doc is not done.