03 — The Asymmetry Rule: W4A16 vs W8A8/FP8, With Worked Roofline Math

Thesis: Decode is memory-bound. Prefill/large-batch is compute-bound. That single asymmetry dictates which bits you quantize where.

Memorize this file. It is the study signal.

1. The rule in one sentence

Quantize weights to fight memory bandwidth (W4A16 for decode). Quantize activations to fight tensor-core flops (W8A8/FP8/FP4 for prefill and high-batch decode).

Everything else is a corollary.

2. Roofline refresher (three numbers, one graph)

Recall from Phase 2. A GPU has:

  • Peak compute (flops/s) at some numeric precision.

  • Peak memory bandwidth (bytes/s) from HBM.

  • Ridge point = peak_compute / peak_bw, in flops/byte. Kernels with lower arithmetic intensity than the ridge are memory-bound; higher, compute-bound.

Real numbers (H100 SXM5, dense, no sparsity):

Precision

Peak (TFLOPs)

HBM BW (TB/s)

Ridge (FLOPs/byte)

FP32 (non-TC)

~67

3.35

~20

BF16/FP16 TC

~989

3.35

~295

FP8 TC

~1979

3.35

~591

INT8 TC

~1979

3.35

~591

B200 (Blackwell, dense):

Precision

Peak (TFLOPs)

HBM BW (TB/s)

Ridge (FLOPs/byte)

BF16 TC

~2250

~8.0

~281

FP8 TC

~4500

~8.0

~563

FP4 TC

~9000

~8.0

~1125

(These are dense numbers. Sparse doubles them on paper.)

RTX 4090 (Ada, for your local rig):

Precision

Peak (TFLOPs)

GDDR6X BW (GB/s)

Ridge (FLOPs/byte)

BF16 TC

~330

~1008

~327

INT8 TC

~660

~1008

~655

FP8 TC

~660 (Ada)

~1008

~655

Key observation: ridge point grows as you drop precision. FP4 on Blackwell has a ridge of ~1125 flops/byte — no LLM operation you can name in decode gets close to that. So FP4 tensor cores are useless for decode unless you also raise the arithmetic intensity (via batching).

3. LLM arithmetic intensity, phase by phase

Decode, batch = 1

Producing one token requires:

  • Loading every weight of every matmul once: 2P bytes of weight traffic (P params × fp16).

  • One vector-matrix multiply per weight matrix: ~2P flops total.

Arithmetic intensity ≈ 1 flop/byte. Miles below every ridge in every table above. Decode at batch 1 is HBM-bandwidth-bound. Full stop. No amount of tensor-core precision helps.

Decode, batch = B

Weights are shared across the batch — you load them once and use them B times:

  • Weight traffic: still 2P bytes.

  • Flops: 2·P·B (B copies of the vector-matrix).

Arithmetic intensity ≈ B flops/byte. To reach the ridge on H100 BF16 (~295 flops/byte), you need B ≈ 295 tokens in flight simultaneously through the same matmul. That’s what continuous batching gets you — not per-user batch, but effective per-matmul batch. In practice at B=32–64 you’re already in the compute-bound regime on most matmuls.

Prefill (any nontrivial length)

Batch = seq_len from the matmul’s perspective. Arithmetic intensity ≈ seq_len. For seq_len > 300, you’re above the H100 BF16 ridge. Prefill is compute-bound.

4. Now the rule, derived

Regime A: Batch 1 decode → W4A16

You’re stuck at intensity ~1 flop/byte, wall = HBM bandwidth.

Your weight matmul does:

time ≈ weight_bytes / HBM_BW
     = (2·P bytes at fp16) / BW

If you quantize weights to INT4 (with per-group-128 scales), weight bytes drop to 0.5·P + overhead ≈ 0.55·P. Time drops proportionally. W4A16 buys you ~3.5× decode speedup at batch 1, essentially entirely from bandwidth reduction.

Activations are still fp16, but there’s only one token’s worth per matmul — negligible traffic. Quantizing activations to INT8 in this regime does nothing because you’re not compute-bound; you just add dequant overhead. Worse than nothing.

Regime B: Prefill / large-batch decode → W8A8 or FP8

You’re above the ridge, wall = tensor-core throughput.

On H100, FP16/BF16 tensor cores do ~989 TFLOPs; FP8 tensor cores do ~1979 TFLOPs — exactly 2×. On B200, FP4 tensor cores do 4× BF16.

So the question in this regime is: what numeric precision does your tensor-core MMA run at? W4A16 kernels dequantize weights to fp16 and run an fp16 MMA — you get zero speedup from weight quantization at high batch (because you’re not bandwidth-bound anymore). W8A8 (or FP8, or FP4) runs the MMA at the lower precision, getting 2×/4× throughput.

This is why W4A16 flatlines at batch 32+ while W8A8/FP8 keep scaling.

5. The worked example

Llama-3-8B (P = 8B, hidden = 4096, kv-heads = 8, head-dim = 128), on H100 SXM (BF16 peak 989 TFLOPs, HBM 3.35 TB/s, ridge ~295):

Batch 1 decode, seq_len irrelevant

  • Weight traffic per token = 2·8B = 16 GB.

  • Achievable time = 16 GB / 3.35 TB/s ≈ 4.8 ms/token → ~210 tok/s theoretical.

  • Measured on real vLLM in bf16: ~150–180 tok/s. Bandwidth utilization ~75–85%, decent.

Now quantize weights to INT4 group-128 (bytes/weight ≈ 0.55):

  • Weight traffic = 8B · 0.55 = 4.4 GB.

  • Achievable = 4.4 / 3350 ≈ 1.3 ms/token → ~760 tok/s theoretical.

  • Measured (AWQ or GPTQ + Marlin): ~450–550 tok/s. ~3× real speedup. Matches the ratio.

Activation quant contribution at batch 1: essentially zero. Correctly predicted.

Batch 64 decode (continuous batching sweet spot)

  • Weight traffic per token: weight_bytes / batch — amortized. At B=64, effective weight traffic per token ≈ 250 MB/token.

  • Arithmetic intensity ≈ 2P·B / weight_bytes = 2·8B·64 / 16 GB = 64 flops/byte. Still below the H100 BF16 ridge of 295 (! that surprised me; check next paragraph). At B=64 we’re actually still weight-bandwidth-limited-ish for MLPs, and compute-limited for attention prefill within the batch.

  • (Correction: for attention, B ≠ compute-bound batch, because KV cache is per-sequence. Only the MLP matmuls benefit from B as arithmetic-intensity multiplier.)

At B=128–256, you’re firmly compute-bound on MLPs. Now the delta:

  • FP16 compute: MLP flops / 989 TFLOPs.

  • FP8 compute: same flops / 1979 TFLOPs → 2× wall reduction.

  • W4A16 gives you nothing here — dequant + fp16 MMA is no faster than fp16 MMA on native fp16 weights.

The crossover point

On H100, empirically:

  • Batch 1–4: W4A16 wins by 2–4×.

  • Batch 4–32: W4A16 still slightly wins because most kernels are still bandwidth-y.

  • Batch 32–128: crossover zone. W4A16 == W8A8 depending on kernel quality.

  • Batch 128+: W8A8/FP8 wins by 1.5–2×.

On RTX 4090 the crossover is later (batch ~64+) because Ada FP8 flops are less advantageous vs bf16.

On B200 the crossover is much earlier for FP4 (batch ~16+) because FP4 tensor cores are 4× faster than BF16 and B200’s compute-to-bandwidth ratio is higher.

6. The corollary map — what to do in each regime

Deployment

Regime

Best choice mid-2026

Local user chatbot, 1 concurrent

Decode-heavy, B=1

W4A16 (AWQ or GPTQ+Marlin), fp8 KV cache

On-prem CRM assistant, 10–100 concurrent

Continuous batching, B_eff ~ 8–32

W4A16 (Marlin/Machete) still often wins; consider FP8 if traffic spikes

Enterprise batch summarization, B=64+

Compute-bound

FP8 (native H100 tensor cores) or W4A8 (QServe) if hardware supports

Blackwell serving fleet

Compute-bound at B=16+

NVFP4 for MLPs (native tensor cores, ~4× BF16), FP8 fallback for attention

Air-gapped Llama on 8×MI300X

Bandwidth-heavy (192 GB/GPU)

W4A16 or FP8; AMD ecosystem improving fast

Speculative decoding drafter (small)

Batch 1 heavy

Draft in FP16 or W4A16 — don’t bother quantizing further

7. The Zoho-specific instantiation

Your on-prem customer has (typical mid-market) 2–4× L40S (48 GB, HBM3 ~864 GB/s, no NVLink between them, BF16 ~181 TFLOPs, FP8 ~362 TFLOPs).

  • Ridge at BF16: 181e12 / 864e9 ≈ 210 flops/byte.

  • Concurrent user count for a chat assistant: rarely above 20 simultaneous active decodes.

  • Effective B per MLP after continuous batching: ~4–16.

Therefore: W4A16 is the correct answer for this hardware. The customer is bandwidth-bound. FP8 doesn’t help until you’re above batch ~64, which they’ll never reach on 4×L40S.

Model sizing: 70B W4A16 with per-group-128 ≈ 40 GB weights + KV. Fits on 2×L40S with room. But 32B W4A16 on 1×L40S gives you more per-user throughput and full replication for redundancy. That’s the real sizing conversation.

Carry this napkin math into every customer call. It’s the differentiator.

8. The one exception — speculative decoding

Spec decoding puts the target model in a compute-bound state even at nominal batch 1 (you’re processing K draft tokens in one forward pass). In principle this raises effective arithmetic intensity to ~K flops/byte, K=4–8.

Still mostly bandwidth-bound at K=8, so W4A16 remains the right target-model choice. But it’s a small nudge, and on Blackwell the argument for FP8-target + fp8-draft becomes credible for the first time.

9. Practice: predict, then measure

Before you leave this file:

  1. Predict on paper: Qwen2.5-32B in W4A16 group-128 on 1×RTX 4090 (24 GB, 1 TB/s), batch 1 decode. Weight bytes = 32B × 0.55 = 17.6 GB. Time/token ≈ 17.6 / 1000 = 17.6 ms → ~57 tok/s theoretical. Real vLLM+Marlin: probably 40–50 tok/s (bandwidth util 70–88%).

  2. Predict Llama-3-70B in W4A16 on 2×L40S (via TP=2), batch 1. Weight per GPU = 35B × 0.55 = 19.25 GB. Time/token ≈ 19.25 / 864 = 22 ms → ~45 tok/s theoretical. Plus all-reduce overhead → real ~30–35 tok/s.

  3. Predict Llama-3-8B in FP8 on 1×H100, batch 128. Now compute-bound. Total flops per iteration ≈ 2·8B·128 = 2 TFlops. FP8 peak ~2000 TFLOPs, achievable 60% = 1200 TFLOPs. Time ≈ 2/1200 s = 1.7 ms per iteration → 75,000 tok/s aggregate (~585 tok/s per user × 128 users). Real measured on well-tuned vLLM: ~50,000 tok/s aggregate. Close.

If your predictions land within 30% of measured, you own this file. Move on.

Next: 04_gptq.md — the actual weight-quantization method that most commonly ships alongside Marlin to make W4A16 work.