07 — The FP8 Family: E4M3, E5M2, and Per-Block Scaling

FP8 is not a research format anymore. It is the current production default on Hopper (H100/H200), the training format DeepSeek-V3 shipped in, and — critically for you — the format your Zoho customers with L40S/L20/H100 fleets will run their production LLMs in. The choice between W4A16 and FP8 is the deployment call.

The two shapes: E4M3 and E5M2

Original spec: Micikevicius et al., “FP8 Formats for Deep Learning,” arxiv 2209.05433 (NVIDIA/Arm/Intel joint, Sep 2022). This is the paper that unified the industry on two variants:

Format

Sign

Exponent

Mantissa

Range (max normal)

Smallest normal

Uses

FP8 E4M3

1

4

3

±448

2⁻⁶ ≈ 0.0156

Weights, activations (better precision)

FP8 E5M2

1

5

2

±57344

2⁻¹⁴ ≈ 6.1e-5

Gradients (better range)

Bit layout of E4M3 (sign S, exponent E, mantissa M):

 S E E E E M M M
[7 6 5 4 3 2 1 0]

value = (-1)^S × 2^(E - 7) × (1 + M/8)     for E ∈ [1..15]     (normal)
value = (-1)^S × 2^(-6)   × (M/8)          for E == 0          (subnormal)
E4M3 uses a special encoding: bias=7, no ±inf, only 1 NaN (S1111111 with M=7).
                    ↑ this is the one deviation from IEEE-754

Why the non-IEEE tweak? E4M3 has so few codes (256 total) that reserving 2×(exponents) worth for NaN/Inf would burn ~1% of the codebook. So E4M3 uses the entire exponent field for normals, giving 448 as the max normal (vs 240 with IEEE-style Inf reservation). E5M2 stays IEEE-compliant because it’s the “gradient” format where Inf propagation is more useful.

study reflex: E4M3 has ~2× more precision than E5M2 at the same magnitude; E5M2 has ~256× more dynamic range. Forward pass wants precision (E4M3); backward pass wants range (E5M2). This is the mixed-precision recipe of Transformer Engine.

Scaling is what makes FP8 usable

FP8 range (E4M3: ±448) is a cliff compared to BF16 (±3.4e38). You cannot just cast tensors to FP8 — most will underflow or saturate. Every FP8 tensor needs a scale factor:

x_fp8   = clip(round(x_bf16 / s), fp8_min, fp8_max)      # quantize
x_hat   = x_fp8 * s                                       # dequant

The tensor-core MMA computes:  Out = (X_fp8 · s_X) @ (W_fp8 · s_W)ᵀ  in fp32 accumulator
                            =  s_X · s_W · (X_fp8 @ W_fp8ᵀ)

The scale gets fused into the fp32 accumulator, so storage is FP8 but math is fp32-accumulated with fp32 scales. This is the whole trick.

The scaling granularity ladder (worth memorizing)

Same shape as the integer granularity ladder from 01, but with FP8-specific defaults:

Granularity

Scale count

Overhead

Where used

Handles outliers?

Per-tensor

1 fp32

4B/tensor

TensorRT-LLM default, simplest

Poorly

Per-channel weight, per-tensor act

1 per output col

Free (fits in bias)

vLLM default for compressed-tensors W8A8-FP8

OK for weights

Per-token activation, per-channel weight

1 per row of X

~0.4% overhead

Modern PTQ (llm-compressor FP8_DYNAMIC)

Good

Per-tile 1×128 activation, per-block 128×128 weight

Many

Small

DeepSeek-V3 training scheme

Excellent

Per-block 32 (MXFP8)

E8M0 shared exp per 32 elems

~3% (2 bits/elem)

OCP MX, Blackwell native

Excellent

The DeepSeek-V3 scheme is worth its own callout because it broke the “fp8 is inference-only” assumption:

DeepSeek-V3: fp8 training in the wild

DeepSeek-V3 technical report, arxiv 2412.19437 (Dec 2024). Trained 671B-total / 37B-active MoE end-to-end in FP8 E4M3, matching BF16 baselines. The scheme:

  • Activations: per-tile scale, tile size 1×128 (one row, 128 columns). Fine enough to isolate outlier channels.

  • Weights: per-block scale, block size 128×128. Coarse enough to be cheap.

  • Accumulator promotion: partial fp32 accumulation on tensor-core, then promoted to full fp32 every 128-K interval (the H100 tensor core internally accumulates in ~fp22; DeepSeek dumps to fp32 periodically to avoid drift).

  • Attention scores stay bf16. The softmax(QKᵀ/√d)V chain is too sensitive; they fp8 only the projections.

  • Optimizer states stay bf16/fp32. FP8 is a forward+backward format, not a state format (see below).

Why this matters to you: DeepSeek-V3 proved that “fp8 is fine for inference but you must train in bf16” is not true. It also proved the scaling granularity has to be finer than per-tensor — this is why Blackwell went native micro-block (MXFP8/NVFP4).

What Hopper actually gives you

H100/H200 tensor cores support FP8 MMA natively:

  • Peak FP8 dense TFLOPS: ~1979 (H100 SXM), ~1979 (H200), 4500 (B200). Compare BF16: 989/989/2250.

  • Ridge point at FP8: ~591 FLOPs/byte on H100 (vs 295 for BF16). This is the number you use in your roofline math from 03_asymmetry_rule.md.

  • Transformer Engine (te.Linear, te.LayerNormLinear) — NVIDIA’s user-facing library. Handles delayed scaling (“history of amax over last N iterations → scale for next iter”), auto-cast policies, fp8 recipe management.

  • cuBLAS exposes fp8 GEMM through cublasLtMatmul with descriptor flags. TensorRT-LLM builds compiled engines that pick fp8 GEMMs.

Peak throughput is not the story though. Because FP8 is 2× smaller than BF16, decode (memory-bound) sees ~2× speedup from bandwidth alone. That’s your Zoho L40S customer’s win.

Where FP8 is winning right now (inference)

  • vLLM --quantization=fp8 — accepts models produced by llm-compressor’s FP8_DYNAMIC recipe (dynamic per-token activation scales, static per-channel weight scales). Zero-calibration, zero-effort quantization for Hopper+.

  • TensorRT-LLM fp8 builds — compiled engine world, higher peak but rigid.

  • SGLang — fp8 KV cache (--kv-cache-dtype fp8_e5m2) as default for long-context serving; near-lossless quality drop vs fp16 KV.

  • DeepSeek-V3/R1 serving — the model ships in fp8 (671B in ~700GB), so this is the only sensible format.

Recent developments <phone_number_or_numberic_id_or_random_id_129> — the FP8-training frontier

You should know these exist because they’re changing the training/inference boundary rapidly:

Paper

ArXiv

One-line takeaway

μnit Scaling (Databricks)

2502.05967

Static scales (no per-iter amax history) work if you initialize network to unit variance. Up to 33% faster than BF16 baseline.

Fully FP8 GEMM Training

2505.20524

Architectural tweaks reduce outliers → all GEMMs including attention projections match BF16.

MOSS FP8 Training

2511.05811

Two-level microscaling + automatic weight scaling to escape amax tuning.

FP8-RL

2601.18150

FP8 rollouts in RL loops via veRL + vLLM/SGLang. Rollout inference is now the bottleneck; fp8 there = ~2× throughput.

The direction of travel is: fp8 is becoming the default across pretrain / SFT / RLHF / inference, with bf16 remaining only for master weights and optimizer states.

Anti-patterns

  • Do not quantize lm_head to fp8. The output projection is small and hyper-sensitive; fp8 on it hurts PPL disproportionately. Leave it bf16.

  • Do not FP8 the attention softmax path. Only the linear projections (q_proj, k_proj, v_proj, o_proj, MLP up/gate/down). QKᵀ, softmax, attention @ V stay bf16.

  • Do not per-tensor FP8 activations on models with massive activations (Llama-3, Mistral-Nemo, etc.). Use per-token dynamic scaling. FP8_DYNAMIC recipe in llm-compressor handles this.

  • Do not benchmark fp8 vs bf16 on Ampere (A100/A6000/3090/4090). Ampere has no fp8 tensor cores — the “fp8 model” runs by dequantizing to bf16 on the fly, which is slower than bf16 native. FP8 wins start at Ada/Hopper (L40S, H100, RTX 4090 does have fp8 but only via slower paths). Blackwell (B200/RTX 5090) is where fp8 really sings.

How to actually produce FP8 checkpoints

# llm-compressor FP8_DYNAMIC recipe — the current standard
from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.transformers import oneshot

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",         # per-channel weight, per-token dynamic act
    ignore=["lm_head"],           # never quantize output projection
)
oneshot(model=model, recipe=recipe)   # no calibration data needed for dynamic
model.save_pretrained("./llama-3.1-8b-fp8-dynamic", save_compressed=True)

Load in vLLM: vllm serve ./llama-3.1-8b-fp8-dynamic --quantization compressed-tensors. On H100 you get ~1.7× decode speedup and ~2× prefill speedup vs bf16, quality within 0.02 PPL on WT2. This is close to a free lunch. On L40S you get ~1.5× decode, similar quality.

The two-sentence study answer

FP8 comes in two shapes — E4M3 for weights/activations (precision-biased) and E5M2 for gradients (range-biased) — always paired with per-tensor/per-channel/per-token or per-block scaling because the ±448 range cliff of E4M3 forces it. On Hopper it’s the near-lossless production default (≈2× decode, ≈2× prefill, <0.05 PPL); DeepSeek-V3 proved it’s viable end-to-end with 1×128 activation tiles and 128×128 weight blocks; Blackwell replaces the coarse per-tensor scaling with native microscaling (see 08).

Homework

  1. Draw the bit layout of E4M3 and E5M2 from memory. Compute the max normal, min normal, min subnormal for each. Confirm E4M3’s max is 448.

  2. Take an fp16 weight tensor from Llama-3-8B (any layer). Compute per-tensor fp8 vs per-channel fp8 vs per-token fp8 quant error. Which channels have the highest error, and where do they land in the model?

  3. Quantize Llama-3.1-8B-Instruct with FP8_DYNAMIC on an H100 or L40S rental hour. Measure decode tokens/sec at batch=1 and batch=32 vs bf16. Report the ratio. This is your first fp8 data point — you’ll cite it in the bake-off.

  4. Read section 3.3 of the DeepSeek-V3 report (the fp8 training section) end-to-end. Note down every fp8 knob they had to add to make training stable — this is the current frontier and it will show up in studies within 6 months.