01 — The Metrics Language of Serving

Before mechanisms, vocabulary. Every engine paper, every optimization post, every SLO conversation with a customer, and every job study at this level of the stack starts by fixing the same six words.

The six numbers you must speak fluently

TTFT — Time To First Token

The wall-clock time from the moment the client’s HTTP request hits the server to the moment the first output token bytes stream back. It is dominated by:

  1. Queue wait (how long the request sat before scheduling)

  2. Prefill compute (the big prompt GEMM)

  3. Tokenization + detokenization overhead (usually small, sometimes surprisingly not — sentencepiece can be slow at extreme lengths)

TTFT is what users perceive as “is this thing broken?” — the responsiveness number. In an agentic pipeline where each turn calls an LLM, cumulative TTFT is a multiplier on every reasoning step. This is the number your Zoho traffic will care about most.

TPOT / ITL — Time Per Output Token / Inter-Token Latency

Two names for essentially the same thing (there is minor pedantry about whether the first-to-second-token gap counts; ignore it and pick a definition per benchmark). This is the steady-state decode cadence — how fast tokens stream once generation begins. Dominated by:

  1. Memory bandwidth (streaming the weights + KV cache each token; the whole reason decode is memory-bound)

  2. Batch contention (batched decode shares GPU across many sequences; bigger batch = more tok/s system-wide but worse ITL per user)

  3. CPU-side scheduling and launch overhead (why CUDA graphs matter)

TPOT is what users perceive as “how fast is it typing?” — the fluency number. Interactive chat wants ~30-60 ms per token (roughly reading speed); batch pipelines can tolerate 200+ ms.

E2E latency — End-to-End

TTFT + (output_len − 1) × TPOT. The full request time. Useful for non-streaming clients but hides the shape.

Throughput — tokens/sec, system-wide

Total output tokens generated by the server per second, summed across all concurrent requests. This is the operator’s metric. Also sometimes reported as prefill throughput (input tokens/sec) and decode throughput (output tokens/sec) separately, because they are fundamentally different regimes.

Goodput — the honest metric

Goodput = throughput restricted to requests that met their SLO. Coined into prominence by the DistServe paper (Zhong et al., OSDI ‘24). The motivating scandal: a system can post enormous throughput numbers by letting p99 TTFT balloon to 30 seconds — nobody would deploy it, but the tok/s chart looks great.

Goodput forces you to state your SLO first (e.g., “TTFT p95 < 500ms, TPOT p95 < 50ms”) and only count tokens that satisfy it. It is the metric that survives contact with a real product manager.

Latency-throughput curve

The fundamental tradeoff plotted: x-axis = throughput (tok/s or req/s), y-axis = latency (TTFT p95 or TPOT p95). As you crank concurrency, both move: more throughput, more latency. The engine’s job is to push this curve down and to the right; the operator’s job is to pick an operating point on it.

Every engine tuning decision is a point on that curve. Bigger max-num-seqs moves you right (throughput) and up (latency). Enabling chunked prefill smooths ITL (down) at slight TTFT cost (up on the y-axis for that metric). Prefix caching moves the whole curve down-and-right when the workload has cache-friendly prefixes.

Percentile discipline (never post means)

Always report p50 / p95 / p99. Means lie systematically for tail-heavy distributions, which every LLM latency distribution is (a handful of requests that hit KV pressure, preemption, or extra-long generation drag the mean toward the tail while the median stays healthy).

Rule of professional identity: you never post a latency number without a percentile. Not in a Slack channel. Not in a benchmark repo. Not in a blog post. study partners listen for this.

Arrival processes and length distributions

Benchmarks that use fixed inter-arrival times and fixed prompt/output lengths are toys. Real workloads are:

  • Poisson arrivals — requests arrive with exponentially distributed inter-arrival gaps at some rate λ (req/s). This creates realistic queueing (bursts and troughs) which is where p99 lives.

  • ShareGPT length distribution — the de facto standard for prompt + output lengths in benchmarks. It has a long tail on both axes, which stresses KV cache management and prefill/decode co-scheduling.

  • Multi-turn traces — for agentic workloads (yours), the sequence of turns per session matters, because it exposes prefix caching behavior that a single-turn benchmark cannot.

Every serious benchmark you run in this phase should specify all three: arrival process, length distribution, concurrency.

How to actually run vllm bench serve

Since V1 became default, the built-in benchmark CLI is the canonical way.

1. Start the server

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-prefix-caching \
  --max-num-seqs 256 \
  --max-num-batched-tokens 8192

Note the three knobs already: prefix caching on, max concurrent sequences, and the token budget per step (chunked prefill’s knob). Every future experiment varies these.

2. Run the benchmark against ShareGPT with Poisson arrivals

vllm bench serve \
  --backend vllm \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name sharegpt \
  --dataset-path /path/to/ShareGPT_V3_unfiltered_cleaned_split.json \
  --num-prompts 500 \
  --request-rate 4.0 \
  --seed 42 \
  --save-result

Key flags:

  • --request-rate 4.0 → Poisson process with λ = 4 req/s. Use inf for as-fast-as-possible (measures peak throughput but destroys latency numbers — report both regimes).

  • --num-prompts 500 → sample size. 500-1000 is the norm for a stable p99.

  • --dataset-name sharegpt → the ShareGPT length distribution.

  • --save-result → writes a JSON of every request’s TTFT/TPOT/E2E for post-hoc analysis.

3. Read the output

A typical block looks like:

============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  127.3
Total input tokens:                      112904
Total generated tokens:                  67340
Request throughput (req/s):              3.93
Output token throughput (tok/s):         529.1
Total token throughput (tok/s):          1416.3
---------------- Time to First Token ----------------
Mean TTFT (ms):                          182.4
Median TTFT (ms):                        142.1
P99 TTFT (ms):                           612.7
----- Time per Output Token (excl. 1st token) -----
Mean TPOT (ms):                          31.8
Median TPOT (ms):                        28.9
P99 TPOT (ms):                           94.2
---------------- Inter-token Latency ----------------
Mean ITL (ms):                           31.5
Median ITL (ms):                         28.7
P99 ITL (ms):                            89.4
===================================================

(Numbers illustrative, not from your machine.)

Always log: hardware, engine version (vllm --version), model, quant, knobs (max-num-seqs, max-num-batched-tokens, prefix-caching on/off, spec-decoding config), dataset, request rate, seed.

The diagnostic reflex

At the end of this phase you should be able to look at TTFT p99 vs TPOT p99 and immediately name the operating regime:

Symptom

Likely cause

First knob to try

TTFT p50 fine, TTFT p99 huge

Queue depth spikes / long prefill blocking new work

Enable chunked prefill; lower max-num-batched-tokens

TPOT p99 >> TPOT p50

Preemption / KV pressure / long-tail sequences

Raise KV cache blocks or gpu-memory-utilization; lower max-num-seqs

Throughput low, TPOT low, GPU util low

Not enough concurrency; you’re leaving compute idle

Raise max-num-seqs; check batch scheduler admits waiters

TTFT fine on repeat prompts, terrible on cold

Prefix cache miss / no cache-aware routing

Enable prefix caching; add hash-key routing at the LB

Everything fine, tok/s below theory

Launch overhead / eager mode

Enable CUDA graph capture (default in V1 for decode)

The table above is the compressed folklore of the field. Memorize it.

SGLang’s equivalent

python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct \
  --enable-radix-cache

python -m sglang.bench_serving \
  --backend sglang \
  --dataset-name sharegpt \
  --num-prompts 500 \
  --request-rate 4.0

Same vocabulary, same discipline. When comparing engines run identical workloads, identical hardware, identical seeds, and be brutally honest about what you find. A 5% difference is inside noise; a 30% difference usually indicates a config mistake before it indicates an engine difference.

The “goodput” exercise

Do this once, keep the notebook forever. On one model on one GPU:

  1. Sweep --request-rate from 0.5 to inf (say 8 points).

  2. For each rate, record TTFT p95, TPOT p95, throughput.

  3. Plot the latency-throughput curve.

  4. Draw a horizontal line at your SLO (say TTFT p95 = 500ms). The throughput at the intersection is your goodput ceiling on this config.

  5. Now flip one knob (enable prefix caching, or enable chunked prefill, or bump max-num-seqs) and redo. Observe how the curve moves.

That notebook — goodput curves with one-variable-at-a-time knob sweeps — is the artifact of a serious inference engineer. It is also the exact deliverable a hiring manager wants to see.

Anchor references