08 — Sizing Exercises: The Napkin Math That Wins Deals¶
The study question, the customer call, the internal capacity review — they all reduce to the same skill: given hardware and a workload, predict the performance envelope. In one hour, unaided. If you cannot do this you are guessing; if you can, you sound like the platform lead. This doc is the reusable template plus five worked examples on real hardware.
1. The Napkin Formula Sheet¶
Memorize these. They are enough for 90% of sizing conversations.
1.1 Memory¶
weights_bytes = params × bytes_per_param # fp16=2, fp8=1, int4≈0.5
kv_cache_per_token = 2 × num_layers × num_kv_heads × head_dim × bytes
kv_pool_size = (gpu_mem × util) - weights - activations_reserve - framework_overhead
max_concurrent_tokens = kv_pool_size / kv_cache_per_token
max_concurrent_users = max_concurrent_tokens / avg_context_length_expected
1.2 Decode speed (batch 1, single GPU, memory-bound)¶
tokens_per_sec_batch_1 ≈ mem_bandwidth_bytes_per_sec / weights_bytes
This assumes the whole model streams for every token. With TP=N, weights per GPU shrink to weights/N but you eat NCCL cost per layer — roughly 80–90% of the naive division at TP=2, 70–85% at TP=4, 60–80% at TP=8 on well-topologied hardware.
1.3 Prefill speed (compute-bound)¶
prefill_tokens_per_sec ≈ peak_tflops_bf16 / (2 × params) # FLOPs per token forward ≈ 2×params
fp8 doubles that, fp4 doubles it again — IF the model actually runs in that dtype and the kernel is real (not padded emulation).
1.4 Aggregate throughput at large batch¶
At batch large enough for decode to become compute-bound, decode approaches prefill’s compute-bound speed divided by output tokens per request. Most production 70B deployments are memory-bound-decode until batch ~64-128; beyond that, quality of the scheduler and kernel decide it.
1.5 TTFT & ITL¶
TTFT ≈ queue_wait + prefill_time + first_decode_step
prefill_time ≈ prompt_tokens / prefill_tokens_per_sec
ITL ≈ 1 / effective_decode_tokens_per_sec_per_user
1.6 The five hardware numbers to know cold¶
GPU |
HBM |
Bandwidth |
Peak BF16 |
Peak FP8 |
Notes |
|---|---|---|---|---|---|
H100 SXM |
80 GB |
3.35 TB/s |
989 TFLOPs |
1979 TFLOPs |
NVL 940 GB/s |
H200 SXM |
141 GB |
4.8 TB/s |
989 TFLOPs |
1979 TFLOPs |
Same compute, more/faster HBM |
B200 |
192 GB |
8 TB/s |
2.25 PFLOPs |
4.5 PFLOPs |
Blackwell; NVL 1.8 TB/s |
B300 |
288 GB |
8 TB/s |
~2.5 PFLOPs |
~5 PFLOPs |
Liquid-cooled; also FP4 native |
MI300X |
192 GB |
5.3 TB/s |
1.3 PFLOPs |
2.6 PFLOPs |
8-GPU node with Infinity Fabric |
MI325X |
256 GB |
6 TB/s |
1.3 PFLOPs |
2.6 PFLOPs |
HBM3e refresh |
L40S |
48 GB |
864 GB/s |
362 TFLOPs |
733 TFLOPs |
Ada; PCIe; no NVLink; entry inference |
RTX Pro 6000 (Blackwell) |
96 GB |
1.79 TB/s |
~700 TFLOPs |
~1400 TFLOPs |
GDDR7; workstation form; 2x L40S memory |
RTX 4090 / 5090 |
24 / 32 GB |
1.0 / 1.79 TB/s |
165 / ~250 TFLOPs |
330 / ~500 TFLOPs |
Consumer; local-first world |
Use peak numbers as the ceiling. Realistically expect 60–80% of peak in production.
2. Worked Example — The Canonical Customer: 4×L40S¶
“Customer has 4×L40S. Best model + quant + engine, and what SLO can we promise?”
2.1 Hardware budget¶
4 × L40S = 4 × 48 GB = 192 GB VRAM total
4 × 864 GB/s = 3.46 TB/s aggregate bandwidth (per-GPU only if you have parallelism; L40S has no NVLink so TP is over PCIe Gen4 = ~64 GB/s all-reduce ceiling per pair)
No NVLink → TP has real cost. TP=2 or TP=4 on L40S is painful; treat these as four independent replicas first, TP only if the model demands it.
2.2 Model choice via memory¶
Model |
fp16 weights |
fp8 weights |
int4 weights |
Fits per L40S? |
|---|---|---|---|---|
Llama-3-8B |
16 GB |
8 GB |
~4 GB |
fp16 fits |
Qwen 2.5 14B |
28 GB |
14 GB |
~7 GB |
fp8 fits comfortably |
Llama 3.3 32B |
64 GB |
32 GB |
~16 GB |
int4/awq fits per GPU; TP=2 fits fp8 |
Llama 3.3 70B |
140 GB |
70 GB |
~35 GB |
Only int4/awq TP=1 (35GB weights + KV), or TP=4 fp8 |
DeepSeek V3 671B (MoE, 37B active) |
1.3 TB |
670 GB |
335 GB |
Does not fit — wrong shape hardware |
Recommended default: Qwen 2.5 14B FP8 or Llama 3.3 32B AWQ int4, replicated 4x on L40S.
2.3 Two scenarios, both defensible¶
Scenario A — Latency-optimized (four independent replicas, small-medium model)
Model: Qwen 2.5 14B FP8, one replica per L40S, TP=1
Weights per GPU: 14 GB (fp8) + 2 GB activations + 2 GB framework = 18 GB
KV pool per GPU: 48 × 0.9 - 18 = ~25 GB
KV per token (Qwen 14B, 48 layers, 8 KV heads, 128 head_dim, fp8 KV): 2 × 48 × 8 × 128 × 1 = 96 KB/token, ~100 KB rounded
Concurrent-tokens per replica: 25 GB / 100 KB ≈ 250,000
At avg 4k context: ~60 concurrent users per replica
Batch-1 decode speed: 864 GB/s / 14 GB ≈ 61 tok/s theoretical, ~45 tok/s realistic
Aggregate: 4 × 60 users × ~15 tok/s per user under load = ~3600 tok/s cluster throughput
Advertise: TTFT p95 = 800 ms at 2k prompt, ITL p95 = 70 ms (14 tok/s user-visible), sustained 3000-4000 tok/s aggregate, 200 concurrent users.
Scenario B — Quality-optimized (one 32B AWQ across all four via TP=4, only if you must)
Only do this if the customer needs a 32B-quality model and you can eat the TP-over-PCIe penalty. Expect ~50% of the naive-scaled throughput; TTFT gets worse; ITL is dominated by all-reduce latency (5-15ms per layer × N layers = brutal). Recommend against unless the quality delta is proven-mandatory.
2.4 SLO commitment sheet¶
Metric |
Commit |
Basis |
|---|---|---|
TTFT p95 (2k prompt) |
1.0s |
Prefill 2000/362 TFLOPs realistic ~200 ms + queue |
ITL p95 |
100 ms |
14 B fp8 at 864 GB/s ≈ 60 tok/s / mid-batch ≈ 10-15 tok/s user |
Aggregate throughput |
3000 tok/s |
4 replicas × 750 tok/s each |
Concurrent users |
200 |
60 per replica × 4 with headroom |
Availability |
99.5% |
With 2-of-4 minReplicas |
If the customer wants better, they need better hardware (see doc 10).
3. Worked Example — The 8×H100 On-Prem Deployment¶
“Bank wants 70B chat + 8B utility on 8×H100 SXM. What can we promise?”
This is doc 12’s capstone in miniature.
3.1 Hardware¶
8 × H100 SXM = 8 × 80 GB = 640 GB VRAM
Full NVLink domain (NVL 940 GB/s per pair)
Aggregate bandwidth: 8 × 3.35 TB/s = 26.8 TB/s (used per-GPU)
3.2 Allocation¶
70B chat: TP=4 on 4 GPUs, one replica. FP8 weights = 70 GB total → 17.5 GB/GPU. KV pool per GPU: 80 × 0.9 - 17.5 - 4 = ~50 GB → 200 GB cluster KV pool.
8B utility: 4 replicas, TP=1, one per remaining GPU. FP8 weights = 8 GB. KV pool per GPU: 80 × 0.9 - 8 - 2 = ~62 GB.
3.3 Numbers¶
70B replica (TP=4, fp8):
Batch-1 decode: 3.35 TB/s / 17.5 GB per shard × TP efficiency 0.75 ≈ 140 tok/s
Under load (batch 32-64): compute-bound territory, aggregate ~2500-4000 tok/s
KV per token (Llama-70B, GQA 8, 80 layers, 128 head, fp8): 2 × 80 × 8 × 128 × 1 = 160 KB/token
Total concurrent tokens: 200 GB / 160 KB = 1,250,000 → at 8k context, ~150 concurrent users
8B replicas (4 × TP=1, fp8):
Batch-1 decode: 3.35 TB/s / 8 GB ≈ 400 tok/s theoretical, ~280 realistic
Aggregate at batch ~50 per replica: ~2000 tok/s × 4 = 8000 tok/s
KV per token (Llama-8B, 32 layers, 8 KV heads, 128 head, fp8): 2 × 32 × 8 × 128 × 1 = 64 KB/token
Per-replica concurrent tokens: 62 GB / 64 KB = ~1M → at 4k context, ~250 users per replica × 4 = 1000
3.4 SLO sheet¶
Route |
Metric |
Commit |
|---|---|---|
70B chat |
TTFT p95 (4k prompt) |
1.5 s |
70B chat |
ITL p95 |
60 ms (16 tok/s) |
70B chat |
Concurrent |
100 users |
8B utility |
TTFT p95 (2k prompt) |
400 ms |
8B utility |
ITL p95 |
30 ms (33 tok/s) |
8B utility |
Concurrent |
800 users |
With prefix caching on for agentic workloads, expect TTFT to improve 3-5x on cache hits. Advertise the cache-cold number; deliver the cache-warm one.
4. Worked Example — The MI300X Alternative¶
“Customer got a deal on 8×MI300X. Can we serve 70B chat there?”
4.1 Why this is interesting¶
8 × MI300X = 8 × 192 GB = 1536 GB VRAM. Enormous.
Bandwidth 5.3 TB/s per GPU — higher than H100.
ROCm + vLLM works; SGLang works; not as battle-tested as CUDA, but real. AMD has been shipping fixes at a rapid pace.
4.2 The 70B on one MI300X¶
A fp8 70B weighs 70 GB. A single MI300X can hold the full model plus ~120 GB of KV pool. That is 750,000 concurrent tokens on ONE GPU. At 8k context, ~90 concurrent users per replica × 8 replicas = 720 concurrent users of 70B on this box without TP overhead at all. Compare to 100 concurrent on the H100 box.
4.3 The catch¶
FP8 kernels on MI300X are less mature than H100 as of 2026. Prefill speed matches H100; batched decode occasionally trails by 10-25%.
Not every quantization scheme has a Marlin equivalent. Test AWQ, GPTQ, FP8 across your workload before committing.
Software risk: fewer engineers debug ROCm than CUDA. Budget for it.
4.4 When MI300X wins¶
Memory-hungry workloads (long-context agentic, 100k+ token histories)
Cost-optimized deals ($1.85/hr Vultr vs $2-3/hr H100 neocloud)
Customers with existing AMD relationships (see Character.AI MI325X case study, doc 05)
Rule of thumb: if the workload’s bottleneck is KV capacity, MI300X/MI325X wins by memory. If it is peak FP8 compute at low latency, H100/H200 wins by ecosystem maturity.
5. Worked Example — The “Just Give Me A Number” study Question¶
“How many tokens per second does Llama-3-70B do on one H100?”
Walk through it aloud:
“Fp16 70B does not fit on one H100 — 140 GB > 80 GB. So either quantize to fp8 (~70 GB, tight, KV starves) or TP across multiple GPUs. Let me assume fp8 with KV cache room — realistic on one 80 GB card only if you keep max_model_len modest.”
“Batch 1 decode is memory-bound. Bandwidth 3.35 TB/s, weights 70 GB → 3.35e12 / 70e9 = ~48 tok/s theoretical.”
“Realistic with kernel and framework overhead ≈ 35-42 tok/s.”
“At batch 32 with prefix caching hot and continuous batching, you can reach ~1000-1500 tok/s aggregate on that same GPU — but with worse per-user ITL (~50 ms → 20 tok/s user-visible).”
Answer arrives with the reasoning; the reasoning is the study signal. The number alone is worth nothing.
6. Worked Example — Scaling An Agentic Workload¶
“Zoho CRM agent traffic: 500 users at peak, average session 20 turns, system prompt + tool defs = 6k tokens, per-turn user message 200 tokens, tool result 500 tokens, model response 300 tokens. Model: Qwen 2.5 14B fp8 with prefix caching.”
6.1 Per-turn arithmetic¶
Turn N prompt = fixed 6k prefix + previous (200+500+300)×(N-1) turns growth + this turn’s user message
At turn 20: 6000 + 19 × 1000 + 200 ≈ 25,200 tokens
Without prefix caching: prefill 25k tokens per turn × 20 turns = 500k prefill tokens per session, all wasted on repetition
With prefix caching (75% hit rate on the growing prefix): effective new prefill per turn ~250 tokens (the diff since last turn) plus a rare full-prefix compute
This is a 100x prefill compute reduction on agent traffic
6.2 Capacity¶
Qwen 2.5 14B fp8 on L40S: KV per token 100 KB (from example 2)
Session KV at turn 20: 25,200 × 100 KB = 2.5 GB per session-in-flight
25 GB KV pool per replica → ~10 concurrent long-session-in-flight per replica
But: with a properly implemented host-memory KV offload (à la Character.AI) between turns, you can hold the session cache on host RAM (192 GB DDR5 easily), and only pay GPU KV cost during the active turn’s decode
With that trick: 60+ concurrent sessions per replica, only ~10 concurrent active decodes
6.3 The Zoho payoff¶
The difference between the naive read (10 concurrent sessions per replica) and the optimized read (60+) is exactly the technical work that justifies your existence on this project. When you ship this as “we can serve 500 agent sessions on 8 L40S”, you have replaced a $80k/year API bill with $16k/year of amortized hardware. Nobody else in the room has done this arithmetic. Make sure your VP is in the room when you present it.
7. The Napkin-Math Discipline¶
Always compute both numbers: the theoretical peak and the realistic 60-80% derating.
Always distinguish batch-1 (memory-bound) from batched (compute-bound at high enough batch).
Always separate prefill from decode. The whole ecosystem’s insight is that they behave differently.
Always include KV cache in memory budget. Weight-only sizing is the amateur mistake.
Always add a 20% headroom — real workloads are lumpier than the model.
Always confirm with a real benchmark within a week. If measured is >20% off from predicted, you learned something — write it down (this is the whole point of the roadmap’s lab notebook discipline).
8. Reading list¶
kipply, “Transformer Inference Arithmetic” — the classic post; internalize it
Horace He, “Making Deep Learning Go Brrrr From First Principles” — the roofline gospel
Character.AI blog on Kaiju — real numbers for a real 20K-QPS system
DeepSeek-V3 report — inference economics section is a masterclass
The Ultra-Scale Playbook by HuggingFace — explains why TP scaling drops with GPU count
9. Exit test¶
Given a random open-weight model config file and a target GPU spec, produce a full sizing sheet (concurrent users, TTFT, ITL, aggregate throughput) in under 20 minutes, no reference material, and confirm within 30% via benchmark.
Explain why the same 70B model on 8×H100 vs 8×MI300X yields different concurrency envelopes and pick the winner for a given workload profile.
Compute the concurrency delta on the 500-user Zoho agent workload with and without prefix caching, and translate it into a dollar figure for a business audience.
Present the 4×L40S example to a customer as a one-page proposal with committed SLOs, budgets, and monitoring plan.