05 — Observability for LLM Inference¶

“You can’t fix what you can’t see” is banal for web services. For LLM inference it’s a doctrine, because the failure modes are counterintuitive (a request is “slow” but might be blocked on queue, prefill, decode, network, tokenizer, or the model itself — and the interesting information about which one lives across 4 different systems). Your Zoho background in long-running services means you probably already have Prometheus, Grafana, OpenTelemetry, and some form of trace store. This doc is about turning that generic stack into an LLM-aware one.

1. The observability triangle for LLMs¶

Three layers, each answering a different question:

Layer	Answer	Tool	Cardinality
Metrics	How is the fleet doing right now?	Prometheus + Grafana	Low (aggregates)
Traces	Where in the pipeline did this specific request spend time?	OpenTelemetry + Tempo/Jaeger	High (per-request)
Logs	What did each component actually say?	Loki, ELK, or equivalent	Very high

Most teams have metrics. Fewer have per-request traces. Almost none have them stitched to the engine internals (schedule time, prefill time, decode time). Owning the traces is the differentiating move.

2. Prometheus metrics from vLLM — the actual metric names¶

vLLM exposes a /metrics endpoint (Prometheus text format) on the same port as the OpenAI API. All metrics prefixed vllm:. Verified against 2026 releases; verify against your exact version because names have shifted.

Engine state / capacity¶

vllm:num_requests_running (gauge) — sequences currently in the running batch.
vllm:num_requests_waiting (gauge) — sequences admitted but not yet started. Primary autoscaling signal.
vllm:num_requests_swapped (gauge) — preempted sequences swapped to CPU. Non-zero = capacity pressure.
vllm:gpu_cache_usage_perc (gauge, 0–1) — KV block pool utilization. Approaching 1 = degradation imminent.
vllm:cpu_cache_usage_perc (gauge) — CPU swap KV utilization.

Throughput¶

vllm:prompt_tokens_total (counter) — total prompt (prefill) tokens processed.
vllm:generation_tokens_total (counter) — total generated (decode) tokens.
Compute rates via PromQL rate(vllm:generation_tokens_total[1m]).

Latency (histograms — use quantile queries)¶

vllm:time_to_first_token_seconds (histogram) — TTFT.
vllm:time_per_output_token_seconds (histogram) — ITL.
vllm:e2e_request_latency_seconds (histogram) — full request.
vllm:request_queue_time_seconds (histogram) — queueing time only.
vllm:request_prefill_time_seconds (histogram) — pure prefill time.
vllm:request_decode_time_seconds (histogram) — pure decode time.

Sequence shape¶

vllm:request_prompt_tokens (histogram) — prompt length distribution.
vllm:request_generation_tokens (histogram) — completion length distribution.
vllm:request_success_total (counter, labeled by finish_reason) — how did requests end.

Cache and preemption¶

vllm:gpu_prefix_cache_hit_rate (gauge or histogram, version-dependent) — the single most important agentic-workload metric.
vllm:num_preemptions_total (counter) — total preemptions. A non-zero derivative is an alarm.

Speculative decoding (if enabled)¶

vllm:spec_decode_num_accepted_tokens_total (counter).
vllm:spec_decode_num_draft_tokens_total (counter).
Acceptance rate = ratio. Below 60% you’re losing money on the drafter.

Known gotchas¶

Not all metrics are on by default. vLLM 2026 gates some behind --enable-metrics or --otlp-traces-endpoint. Verify with curl :8000/metrics | grep vllm:.
Metric names have changed between releases. Historical shifts: vllm:e2e_request_latency_seconds was once vllm:request_e2e_latency_seconds; vllm:gpu_cache_usage_perc was once vllm:gpu_cache_usage_sys_perc. Always run curl :8000/metrics | head -100 on your actual version.
Process-level metrics (process_virtual_memory_bytes, process_resident_memory_bytes) are CPU-only. vLLM does NOT expose per-process GPU memory via Prometheus. Use DCGM Exporter for GPU-level metrics (see §3).
Reference docs: docs.vllm.ai/en/stable/design/metrics. Includes a reference Grafana dashboard.

3. GPU-level metrics via DCGM Exporter¶

vLLM metrics tell you about the engine. DCGM Exporter (NVIDIA’s Data Center GPU Manager exporter) tells you about the GPU itself:

DCGM_FI_DEV_GPU_UTIL — raw GPU util (see 04_autoscaling_llms.md §1 for why not to autoscale on this, but still useful for postmortem).
DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_FREE — GPU memory used / free.
DCGM_FI_DEV_MEM_COPY_UTIL — memory bandwidth utilization %.
DCGM_FI_DEV_TENSOR_ACTIVE (Hopper+) — tensor-core activity.
DCGM_FI_DEV_POWER_USAGE (Watts) — power draw.
DCGM_FI_DEV_GPU_TEMP / DCGM_FI_DEV_MEMORY_TEMP — thermals.
DCGM_FI_DEV_ECC_SBE_VOL_TOTAL / _DBE_ — ECC errors. Non-zero double-bit errors = failing GPU; alert immediately.

Deploy via NVIDIA GPU Operator (which includes DCGM Exporter as a DaemonSet).

The joint dashboard¶

A useful Grafana dashboard combines vLLM metrics with DCGM. Panels I’ve found essential:

Request queue depth (vLLM) + GPU util (DCGM) side-by-side per replica. Visually reveals the util paradox.
Tensor-core active % — tells you decode vs prefill dominance (prefill is GEMM-heavy, decode is not).
KV cache usage % vs preemption rate — shows the moment capacity binds.
TTFT p50/p95/p99 vs prompt length p95 — correlate long-prompt spikes with TTFT breaches.
Prefix cache hit rate vs mean TTFT — the ROI panel for prefix caching (huge deltas on agent traffic).

Ref examples: akrisanov.com/vllm-metrics for a walk-through with real screenshots.

4. OpenTelemetry: per-request tracing¶

Metrics tell you aggregates. When one customer’s request is slow, metrics won’t help. You need traces.

The trace model for an LLM request¶

A single request produces a trace with (at minimum) these spans:

request (root span)
├── auth
├── router.route
│   └── router.pick_replica  (attributes: cache_hit=true, target_pod=...)
├── engine.enqueue
├── engine.wait               (queue time)
├── engine.tokenize
├── engine.prefill            (prompt_tokens, kv_blocks_allocated)
├── engine.decode             (repeatable event stream: token N, ITL_ms)
│   ├── token.first_emitted   (TTFT boundary)
│   └── token.stream_chunks
└── engine.detokenize

Every span carries useful attributes. A slow request tells its own story: was queue time 800ms? Prefill? Decode? Was cache hit missed unexpectedly?

Instrumentation, concretely¶

vLLM 2026 has native OTel support. Turn it on:

vllm serve ... \
  --otlp-traces-endpoint http://otel-collector:4317 \
  --collect-detailed-traces=all

vLLM emits spans for engine.enqueue, engine.prefill, engine.decode, etc., automatically. Cross-verify the exact flag name against your version.

In your gateway/router, propagate trace context: read traceparent header on incoming request, wrap the downstream engine call in a client span, forward traceparent in your outgoing HTTP.

In your agent harness code, wrap each LLM call in a llm.call span with:

llm.model — the model name.
llm.prompt_tokens, llm.completion_tokens.
llm.tool_calls_count (agentic!).
llm.finish_reason.
llm.temperature, llm.top_p.
Optionally, hashed prompt prefix for cache-behavior debugging (never the raw prompt — PII risk).

Use an OTel Collector to receive OTLP and export to Tempo (Grafana) or Jaeger.

Sampling policy¶

At 20K RPS you cannot afford 100% traces. Options:

Head-based random sampling (10%) — simple, misses tails.
Tail-based sampling (via OTel Collector) — keep 100% of slow / errored requests, sample happy path. This is what you want; tail-based is the correct default for LLM latency debugging.
Adaptive sampling — sample more heavily during incidents. Sophisticated; nice to have.

The killer trace attribute set for agentic workloads¶

agent.session_id — group multi-turn traces into a conversation.
agent.turn_number — which turn we’re on.
agent.tool_name — which tool the LLM decided to call (if any).
llm.prefix_cache_hit — did the KV cache hit? Extracted from vLLM’s per-request metric.
llm.prompt_shape — e.g., "system+history12+tools+user". Free-text tag; makes filtering brilliant.

Given your day job, this is the trace shape you should be championing at Zoho.

5. SLO dashboards — percentiles discipline¶

Never quote means for latency. Ever. Means hide the exact behavior your users experience.

The panels¶

Per model + per tier (interactive / batch):

TTFT: p50 / p95 / p99 — 3-line Grafana panel. Alert on p95 breach.
ITL: p50 / p95 / p99 — same.
E2E latency: p50 / p95 / p99 — same.
Throughput: tokens/sec (rate).
Concurrency: num_requests_running.
Queue depth: num_requests_waiting.
Prefix cache hit rate.
Preemption rate.

PromQL: how to actually get p95 from histograms¶

histogram_quantile(0.95,
  sum by (le, model) (
    rate(vllm:time_to_first_token_seconds_bucket[5m])
  )
)

Common mistakes:

Forgetting by (le) — gives you a single scalar instead of a histogram.
Using avg instead of sum — wrong aggregation for _count bucket counters.
Rate window too short — [1m] on a low-QPS metric gives you zeros. Rule of thumb: rate([<4× scrape interval>]).

SLO framing¶

Don’t just draw lines and hope. Write your SLOs down explicitly:

Service: llama-70b-chat, tier: interactive
  TTFT: p95 < 500ms   error budget: 5% of 30-day window
  ITL:  p95 < 50ms    error budget: 5% of 30-day window
  E2E:  p95 < 8s      error budget: 5% of 30-day window
  Availability: 99.5% of requests receive a final SSE chunk

Have alerts based on error budget burn rate, not raw threshold breaches. Google SRE book §4.

6. CI performance gates¶

Every engine upgrade (vLLM 0.7.3 → 0.7.4) can silently regress performance. Your CI should catch this before prod does.

The perf-gate skeleton¶

Nightly (or on every engine tag), run a benchmark harness against a canary deployment:
- vllm bench serve or SGLang’s benchmark, or a custom driver using your real prompt shape.
- Realistic arrival process (Poisson at ~80% of prod QPS).
- Realistic prompt length distribution (sampled from prod, PII-scrubbed).
Record TTFT/ITL/throughput/goodput.
Compare against the last N runs’ baseline.
Fail CI if:
- TTFT p95 regressed > 5%.
- Throughput regressed > 3%.
- Goodput (tokens/sec meeting SLO) regressed > 5%.
Post a report to Slack/PR-comment either way.

The Zoho play¶

Build this harness as the internal LLM benchmark harness and own it. From 13_zoho_leverage_plan.md:

Every org needs one, few have one, and it makes you the arbiter of engine/model/quant decisions.

The person who owns the benchmark is the person who owns the decision. That’s a career-shaping asset.

7. Logging (briefly)¶

vLLM logs to stdout. Route via your DaemonSet (Fluent Bit / Vector) to Loki/ELK.

Essential log fields to enrich:

request_id (matches trace ID).
model_name.
pod_name, node_name.
Never log full prompts. PII/security disaster. Log token counts, hashed prefix, finish reason, latency, error class.

Log levels:

INFO: request received, request completed.
WARN: preemption, KV pool full, throttle applied.
ERROR: OOM, engine crash, model load failure.

8. What the Character.AI / Anthropic / DoorDash / Character-of-your-shop blogs teach you¶

Character.AI’s blog series (blog.character.ai) is the current gold standard for LLM inference observability transparency. Read it not for the specific numbers (they’ll date) but for the categories of things they instrumented:

20K QPS, 33× cost reduction since 2022, 13.5× cheaper than commercial-API-built alternatives, sub-1-cent-per-hour-of-conversation.
Metrics on cross-layer KV sharing, host-memory KV cache retention across chat turns, sliding-window attention 5:1 SW:global ratio.
Then the AMD deal: same workload on MI325X delivered 2× QPS while holding p90 TTFT and p90 TPOT under target — look at how they measured those; that measurement discipline is the point.

Other worthy reads: Anthropic’s engineering blog, DoorDash engineering, LinkedIn’s LLM serving posts. The pattern is always: honest metrics, percentile discipline, and specific line-item improvements with before/after numbers.

Reading list¶

vLLM metrics reference — docs.vllm.ai/en/stable/design/metrics + the reference Grafana JSON.
akrisanov.com/vllm-metrics — practical monitoring walkthrough.
Google SRE book, chapters on SLOs and error budgets.
OpenTelemetry semantic conventions for GenAI — they’ve standardized gen_ai.* attribute names in 2025–26; use these.
Character.AI blog series on inference optimization.

Exit test¶

Recite the top 8 vLLM Prometheus metric names and what each means.
Write the PromQL for TTFT p99 per model over 5-min window. From memory.
Design the trace attribute set for an agent tool-calling request going through your gateway.
Explain error budget burn-rate alerting to a colleague. Under 3 minutes.