Blackwell NVFP4 Is a Training Recipe, Not a Four-Bit Switch

A team lowers numeric precision, sees memory use fall, and expects training to get cheaper. Then loss curves wobble, gradients become fragile, or the final model misses a quality target. Four-bit training is not a storage-format toggle: it changes how the training loop represents a wide, moving range of numbers. NVIDIA’s Blackwell NVFP4 stack is interesting because it treats this as a complete numerical recipe.

The four-bit value is only one part of the number

NVFP4 uses an E2M1 value: one sign bit, two exponent bits and one mantissa bit. Its representable magnitude tops out at six, so it cannot directly encode the full range of typical tensors. Transformer Engine reconstructs a value using a hierarchy: the four-bit value is multiplied by an FP8 E4M3 scale shared across a block of 16 values, then by an FP32 scale for the tensor. A local scale follows neighborhood variation while the global scale maps the tensor into the format’s range. (Transformer Engine documentation)

The block is not bookkeeping. If one unusually large value sets the scale for a whole tensor, smaller values can collapse toward zero. More local scales can preserve variation, but the scales themselves need storage and computation, and their layout must suit the matrix operations that consume them. Transformer Engine uses two-dimensional 16-by-16 scaling for weights by default, while activations and gradients use one-dimensional blocks. The stated aim is to keep rowwise and columnwise weight views numerically consistent.

Stability comes from coordinated choices

Rounding every value to its nearest representable level can introduce systematic bias. For gradients, Transformer Engine supports stochastic rounding: values are probabilistically rounded to adjacent representable values so that the expected rounded value matches the original. This does not make an individual gradient exact; it avoids repeatedly pushing rounding error in one direction.

A second tool addresses outliers. A random Hadamard transform rotates selected tensors before quantization, spreading concentrated extreme values across dimensions. NVIDIA applies it to columnwise quantization of inputs and gradients used for weight-gradient matrix multiplication, which the authors identify as especially sensitive. The transform costs compute, so the benefit depends on the error it prevents.

The recipe also keeps selected layers or operations at higher precision. That is not a failure of FP4. If a small number of numerically sensitive components govern convergence, preserving them may be more efficient than widening every tensor.

NVIDIA’s paper describes these techniques together and reports a 12-billion-parameter model trained on 10 trillion tokens with loss and downstream accuracy comparable to its FP8 baseline. The study is authored by NVIDIA researchers; it is evidence for that model, data and configuration, not an independent replication or a proof for every training run. (paper)

A format does not tell you whether the workload wins

Low precision can reduce operand bandwidth and let specialized Tensor Cores perform more arithmetic. But a training step also includes scale calculation, conversion, optimizer state, communication, activation storage and synchronization. A matrix multiply that becomes faster may not dominate the full step.

Nor should a training result be projected onto inference. A recent independent Blackwell attention-kernel preprint reports that fully FP4 attention can lose time to quantization after softmax and data movement, and proposes a mixed QK4/PV8 path. It studies inference kernels, not pretraining, but illustrates the principle: multiplication helps only if conversion, scaling and memory movement do not consume the savings. Its measurements are specific to named models and contexts. (MpFA paper)

What a training team should validate

Compare an agreed baseline and recipe at equal training progress, not just equal wall-clock snapshots. Record model and data configuration, software versions, hardware, precision settings, scaling and higher-precision exceptions. Inspect loss and gradient behavior over the full run, held-out task quality, step time including conversion and communication, total memory use, and reproducibility after restart. Do not compare only peak Tensor Core throughput or infer quality from matching loss.

Teams interested in exact replay should treat hardware, software and operation order as part of the experiment; our guide to deterministic training explains why a seed alone does not guarantee identical states.

There is also a distributed-training constraint: columnwise quantized tensors gathered across ranks need a shared global scale. Scale metadata must be computed, synchronized and laid out in the form the GEMM expects. In other words, a supported format can still require coordination in the data-parallel pipeline. A useful pilot first checks a few representative layers and activation distributions, measures error against a higher-precision reference, then runs long enough to detect whether local deviations accumulate into a convergence or quality difference. A short synthetic GEMM benchmark cannot answer that last question.

The advance is not that four bits are safe everywhere. It is that a low-precision format can become viable when representation, scaling, rounding, outlier treatment, layouts and selective precision are designed together. That makes the implementation more complex and the claim more bounded. Keep a higher-precision control, define quality gates before the run, and measure the entire training step. The evidence belongs to a recipe and a workload—not to the label “FP4” by itself.