08 — FP4, MXFP4, NVFP4: The Blackwell Frontier¶

This is the format war that is happening right now in <phone_number_or_numberic_id_or_random_id_130>. If you understand this file cold you will be ahead of ~95% of “AI engineers” on LinkedIn.

Why 4-bit floats at all?¶

Int4 works fine for weights (GPTQ, AWQ) because weights are bell-shaped and small groups (g=128) tame the outliers. Int4 fails on activations because activation channels have 100,000× dynamic range. Enter FP4: give up mantissa (only 1 bit of precision!) to buy back exponent range.

Bit layout of FP4 E2M1 — the format everyone converged on:

 S E E M
[3 2 1 0]

Codepoints (16 total, only 15 unique due to ±0):
  { 0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, ±6 }

Max normal = 6.0

Read that codepoint list until it’s obvious. That is the entire alphabet of your weights and activations in a FP4 world. Everything has to snap to one of those 15 values. This is why the scale factor becomes the whole ballgame — it decides how the 15-value palette maps onto real numbers.

The two Blackwell formats: MXFP4 vs NVFP4¶

Here is the single most important table in this file. Memorize it.

Format	Group size	Block scale format	Block scale storage	Extra top-level scale	True bits/weight	Origin
MXFP4	32	E8M0 (power-of-2 only)	8 bits per block	None	4 + 8/32 = 4.25	OCP Microscaling spec (2023, industry-wide)
NVFP4	16	FP8 E4M3 (full float)	8 bits per block	FP32 per-tensor	4 + 8/16 + tiny = ~4.5	NVIDIA Blackwell whitepaper (2024)

The implications:

MXFP4 group size 32 is coarser, and its E8M0 scale (which is a power of 2, i.e. only exponent, no mantissa) can only stretch/shrink the codebook by factors of 2. If your block’s max is 3.7, the closest E8M0 scale rounds you to either max=4 (waste) or max=2 (saturate). ~2× waste on average vs a full FP scale.
NVFP4 group size 16 is 2× finer, and its FP8 E4M3 scale has 3 bits of mantissa — continuous enough that saturation is negligible. Plus a second per-tensor FP32 scale on top to keep the block scales themselves in a good range. This is a two-level microscaling scheme.
The bits/weight difference (4.25 vs 4.5) is tiny in storage. The accuracy difference is not tiny.

The 2025 finding that changes the story: MR-GPTQ¶

“Bridging the Promise vs Performance for MXFP4” — IST-DASLab / Neural Magic, arxiv <phone_number_or_numberic_id_or_random_id_124> (Sep 2025). This paper is the current map of the FP4 landscape and you should read the abstract at minimum.

Key diagnoses:

NVFP4’s tiny group size (16) neutralizes outlier-mitigation methods. SmoothQuant, AWQ, rotation methods — all designed for group=128 — lose most of their advantage when the group is already small enough to isolate outliers on its own. In NVFP4 land, you don’t need clever tricks; the format does the work.
MXFP4’s power-of-two scale is a real accuracy hit. The E8M0 quantization of the scale itself introduces error that clever weight/activation methods can’t recover from.
MR-GPTQ (Multi-Resolution GPTQ): block-Hadamard rotation + format-specific GPTQ solves both. Result: 3.6× layer speedup / 2.2× E2E speedup on B200 and 6× layer / 4× E2E on RTX 5090 vs FP16 baselines, at near-lossless quality.

The study reflex: NVFP4 is easier to use, MXFP4 needs help. On Blackwell, if you can pay for the FP8 scale storage, NVFP4 is the current sweet spot. On non-Blackwell hardware emulating FP4, MR-GPTQ closes the MXFP4 gap.

Why Blackwell native matters (the roofline argument you already know)¶

B200 tensor-core peak (dense, per the whitepaper):

Format	TFLOPS	Bytes/param	Ridge point
BF16	2250	2	281
FP8	4500	1	563
FP4 (MXFP4/NVFP4)	9000	0.5	1125

A 70B model in FP4 is ~35 GB. That fits on one B200 (192GB HBM3e) with room for 90+ GB of KV cache and activations. Decode speedup vs BF16: ~4× from bandwidth alone. Prefill speedup: 4× from compute. These are the numbers that will drive the next H100→B200 procurement cycle.

Zoho translation: your customers with H100 fleets today will be looking at B200/GB200 refresh in 2026-2027. Being fluent in FP4 semantics now is the leverage point.

The other Blackwell-adjacent papers you should recognize¶

Paper	ArXiv	One-line
Training LLMs with MXFP4 (Tseng, Yu, Park)	<phone_number_or_numberic_id_or_random_id_125>	Stochastic rounding + Hadamard on activations → near-lossless MXFP4 training up to 6.7B, >1.3× vs FP8.
Native FP4 Pretraining on MI355X	<phone_number_or_numberic_id_or_random_id_126>	AMD Instinct MI355X does native MXFP4. Weight-gradient (wgrad) is the primary divergence source; deterministic Hadamard rotations stabilize.
Diagnosing FP4	<phone_number_or_numberic_id_or_random_id_127>	MLP up/down projections dominate FP4 sensitivity. Under MXFP4, early blocks can be more sensitive than late blocks (reverses the intuition).
TORQ	<phone_number_or_numberic_id_or_random_id_128>	Two-level orthogonal rotation for MXFP4 activations. Fixes inter-block variance imbalance + codebook collapse.
MixFP4	<phone_number_or_numberic_id_or_random_id_131>	Per-block choice of E2M1 vs E1M2, cleverly stored in the sign bit of E4M3 scale.
SharQ	<phone_number_or_numberic_id_or_random_id_132>	FP4 + N:M sparsity dual-path GEMM on RTX 5090. 2.2–2.4× vs FP16.
OAS/MBS for MXFP4	<phone_number_or_numberic_id_or_random_id_133>	Overflow-Aware Scaling + Macro Block Scaling. Closes MXFP4→NVFP4 gap from ~10% to <1%.
DuQuant++ (MXFP4 variant)	<phone_number_or_numberic_id_or_random_id_134>	Extends DuQuant rotation family to native g=32 microblocks.

The pattern: almost every paper in this space is either “how to rotate around the group boundary” or “how to compensate for the scale-format restriction.” Once you see that, you don’t need to read all of them — just remember which failure mode each addresses.

The OCP Microscaling (MX) spec — the underlying standard¶

Open Compute Project (OCP) published the MX v1.0 spec in Sep 2023. It defines a family of block-scaled formats sharing one recipe:

Block size: always 32 elements.
Block scale: always E8M0 (8-bit power-of-2 exponent, no sign, no mantissa). Range 2⁻¹²⁷ to 2¹²⁷; also encodes NaN.
Element formats: MXFP8 (E4M3 or E5M2), MXFP6 (E3M2 or E2M3), MXFP4 (E2M1), MXINT8.

MXFP8 (with group=32) is a serious competitor to plain FP8 with per-tensor scaling — for the same storage cost you get much better outlier handling. Blackwell supports MXFP8 natively; expect MXFP8 to increasingly displace plain FP8 for inference over the next 1-2 years.

NVFP4 breaks the spec by using group=16 and an FP8 (not E8M0) scale plus a top-level tensor scale. That’s why NVIDIA had to name it separately — it is not OCP MX-compliant. NVIDIA is betting that the accuracy win justifies the format fragmentation.

Where FP4 is actually available today¶

Hardware	Native FP4?	Format	Status
B200/B100	Yes	NVFP4 + MXFP4	Shipping, 2024+. Production.
GB200 NVL72	Yes	Same as B200	Shipping, 2024+. Production frontier training.
RTX 5090 (Blackwell consumer)	Yes	NVFP4 + MXFP4	Shipping late 2024. r/LocalLLaMA is all over this.
RTX 4090	No	Software emulation only	Skip — use int4 (Marlin) instead.
H100/H200	No	Software emulation	Skip — use FP8 (native) instead.
MI355X (AMD Instinct)	Yes	MXFP4 native	Shipping 2025. Under-covered but real.
MI300X	No	—	FP8 native, no FP4.

The single strongest reason to care about FP4 right now: RTX 5090 puts native FP4 tensor cores in the hands of the r/LocalLLaMA community. Within 12 months there will be a full ecosystem of 5090-optimized FP4 quantization work — be one of the people who read the papers first.

Where FP4 breaks (still, in <phone_number_or_numberic_id_or_random_id_135>)¶

Instruction-following degradation shows up before PPL degradation. Naive MXFP4 on Llama-3-70B loses 5-8 points on chat evals while PPL barely moves. Always chat-eval FP4 models (MT-Bench, IFEval).
Long-context damage — quant errors compound in KV cache. FP4 activations + FP4 KV cache at 32k+ context is not yet solved for production quality.
Reasoning traces (o1/R1-class) — the massive-activation problem is worst in models that do explicit chain-of-thought; FP4 there is still risky.
Small models under 3B — FP4 hits them harder than 7B+ because there’s less redundancy in the weights.

Anti-patterns¶

Don’t compare FP4 vs FP16 on WT2 PPL alone. PPL is insensitive to the failure modes that matter. Add chat evals + KL-div.
Don’t use MXFP4 without rotation. Vanilla MXFP4 loses 3-6% on MMLU vs the same-bits AWQ. TORQ, MR-GPTQ, DuQuant close the gap.
Don’t confuse NVFP4 with FP4 generally. When you see “FP4” in a spec sheet, ask which. On Blackwell it’s usually NVFP4; on OCP-aligned hardware it’s MXFP4.
Don’t pay for FP4 on non-native hardware. On Ampere/Hopper, INT4 (Marlin) is faster than FP4 emulation.

Tooling status (verify before you build on this)¶

Tool	FP4 support	Notes
llm-compressor	NVFP4 shipping; MXFP4 in progress	The reference PTQ path for FP4. Ships `NVFP4` and `MXFP4A16` recipes.
vLLM	NVFP4 + MXFP4 via compressed-tensors	Native on B200; emulated elsewhere (skip).
TensorRT-LLM	FP4 via ModelOpt	Blackwell only; part of NVIDIA production stack.
AutoAWQ	Partial NVFP4 experiments	Not yet production-ready.
llama.cpp	Some GGML FP4 tinyllama experiments	Watch this space — CPU/M-series FP4 is coming.

How to actually produce an NVFP4 checkpoint (2026)¶

# llm-compressor NVFP4 recipe
from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.transformers import oneshot

recipe = QuantizationModifier(
    targets="Linear",
    scheme="NVFP4",              # W-NVFP4 + A-NVFP4-dynamic
    ignore=["lm_head"],
)
oneshot(model=model, dataset=calibration_data, recipe=recipe)
model.save_pretrained("./llama-3.1-70b-nvfp4", save_compressed=True)

On B200: expect ~3.5× decode throughput vs BF16, ~4× prefill, MMLU drop <0.5 points on Llama-3.1-70B.

The two-sentence study answer¶

FP4 comes in two Blackwell-native flavors: MXFP4 (OCP-standard, g=32, E8M0 power-of-two block scale, ~4.25 bits/weight) and NVFP4 (NVIDIA-only, g=16, FP8-E4M3 block scale + FP32 tensor scale, ~4.5 bits/weight) — both use the same E2M1 codebook {0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, ±6}. NVFP4’s tiny group size makes outlier-mitigation almost redundant (the format handles it); MXFP4’s coarser E8M0 scale creates a real accuracy gap that MR-GPTQ (block-Hadamard + format-specific GPTQ, arxiv <phone_number_or_numberic_id_or_random_id_136>) is currently the SOTA fix for.

Homework¶

Write down the 15 unique FP4 E2M1 codepoints from memory. Compute the mean and stdev of the codebook. Compare against a normal distribution.
Take one MLP down_proj weight from Llama-3-8B. Quantize it three ways: NVFP4 (g=16, FP8 scale), MXFP4 (g=32, E8M0 scale), INT4-g128 (GPTQ). Compare quant error histograms.
Read the MR-GPTQ abstract + section 4 of arxiv <phone_number_or_numberic_id_or_random_id_137>. Note the specific hardware speedup breakdown on B200 vs RTX 5090 — you’ll want to cite this.
Watch the r/LocalLLaMA feed for “5090 NVFP4” posts through H1 2026. First-mover papers on consumer Blackwell will drop; being the person who read them first is a real edge.