Inference Engineering · §5.1.1

Quantization Granularity

A scale factor maps low-precision values back to high precision. Granularity is how many values share one scale factor. Share too many and a single outlier drags everything else toward zero — which is the entire reason MXFP4 and NVFP4 exist.

0 · What a scale factor does

The mapping

scale = max(|x| in group) / QMAX
q = round_to_format(x / scale)
x̂ = q × scale ← reconstructed

FP4 (E2M1) can only represent eight magnitudes: 0, .5, 1, 1.5, 2, 3, 4, 6. Everything in a group has to land on one of those eight rungs after dividing by the group's single scale factor.

Why the group matters

The scale is set by the largest value in the group. Put one outlier of 9.0 in with a crowd of 0.3s and the scale becomes 1.5 — so every 0.3 rounds to either 0 or 0.75. The outlier didn't just survive, it flattened its neighbors.

"More granular quantization has a lower chance of smoothing over outliers, preserving quality. However, more granularity introduces more overhead for storing and applying scale factors."

1 · Watch it happen — real FP4 quantization

Granularity

Outliers

magnitude |x| quantization error one scale factor per outlined group

Scale factors

—

for 384 values

Relative RMSE

—

error vs. original tensor

Values crushed to 0

—

non-zero inputs that quantized to zero

Scale overhead

—

extra bits per stored value

Real FP4 E2M1 rounding on a synthetic 6-channel × 64-value tensor with a realistic outlier distribution. Numbers recompute live — nothing here is hardcoded.

2 · The three levels

Tensor level

A single scale factor for the entire QKV tensor.

Cheapest. One extra number for millions of values — overhead is effectively zero.
Most fragile. The single worst outlier anywhere in the tensor sets the scale for everything.
Fine for weights, which are well-behaved. Dangerous for activations, which are not.

Channel level

A different scale factor for each feature vector within the tensor.

Contains the blast radius to one channel instead of the whole tensor.
Works especially well because activation outliers tend to concentrate in specific channels — flip the outlier toggle above to see this matter.
Still cheap: one scale per few thousand values.

Block level

Within each feature vector, split into blocks of N values, one scale factor each.

Highest fidelity. An outlier only distorts its own 16 or 32 neighbors.
This is what makes 4-bit viable at all — the format's dynamic range is tiny, so the scale has to adapt locally.
Pays for it in stored scale factors and in compute to apply them.

3 · How real formats pick their block size

Format	Granularity	Scale storage	Effective bits/value
FP8 (E4M3), classic	tensor or channel	1 per tensor / channel	~8.0
MXFP8 · Blackwell	block of 32	8-bit shared exponent	8.25
MXFP4 · Blackwell	block of 32	8-bit shared exponent	4.25
NVFP4 · NVIDIA, Blackwell	block of 16	FP8 scale + 32-bit global	~4.5

Why 4-bit needs microscaling

The book's framing: MX formats "compute a blockwise scale factor on every 32 parameters, reducing the impact of these number formats' lower dynamic range." Granularity is the compensation mechanism for having almost no dynamic range left. NVFP4 goes further — block size 16 plus a secondary 32-bit global scale factor — precisely because 4 bits needs the help.

What it costs

Two costs, not one. Memory: the small-block scale factors have to be stored and read, shaving the compression win. Compute: both the block and tensor scale factors have to be applied on every operation. Blackwell offsets the second by applying scale factors inside the Tensor Cores — which is why these formats are architecture-specific rather than universal.

4 · Where this sits in the bigger decision

Granularity is one of two knobs the book asks you to set after choosing what to quantize. The other is dynamic range — the reason floating-point formats beat integer ones, since the exponent bits let FP represent outliers that INT simply clips. Together they decide whether a quantized model is production-viable.

Model components are not equally forgiving. From least to most sensitive: linear-layer weights → activations → KV cache → attention. The last two carry compounding error — a KV cache entry feeds every subsequent token, and each attention calculation depends on the previous one, so small errors accumulate over thousands of tokens. Which is why softmax stays in original precision in all but the most aggressive schemes.

Source: Inference Engineering §5.1–5.1.2 · quantization computed live with FP4 E2M1 rounding