Why Blackwell Attention Can Hit the Softmax Wall Before FP4 Peaks

The GPU’s fastest engine is not always the part holding a workload back. Long-context attention surrounds matrix operations with reductions, exponentials, scaling, layout changes and transfers between memory levels. A new paper about Blackwell makes this tension concrete: pushing every operand to FP4 can move the bottleneck into the steps required to feed and stabilize the matrix multiply.

Attention is a pipeline, not one matrix multiply

For each attention head, a kernel forms query-key scores, applies scaling and softmax normalization, then combines probabilities with values. Matrix operations fit Tensor Cores, but softmax has dependencies: the kernel must compute row statistics, normalize scores and preserve numerical behavior while moving tiles through the GPU.

Lower precision can reduce operand traffic and increase arithmetic throughput. It does not remove the softmax reduction or the need to convert and scale values. If a kernel repeatedly materializes intermediate tensors, it may spend bandwidth moving them. If conversion happens in the hot loop, it can erase part of the theoretical multiply advantage. The balance changes with sequence length, head dimensions, batch size and tile layout.

NVIDIA’s NVFP4 training format uses four-bit E2M1 values with local and global scales, and Transformer Engine documents additional transformations for training. Those details do not automatically transfer to a hand-written inference kernel: each kernel has its own layouts, conversion costs and accuracy requirements. (Transformer Engine NVFP4 guide)

MpFA chooses different precision for the two products

MpFA proposes QK4V8: use NVFP4 for the query-key score product and FP8 for the probability-value product. The authors say fully FP4 attention can pay for online quantization after softmax, tensor/shared-memory movement and contention around the softmax path. Keeping the second product in FP8 avoids some conversion and scaling overhead, according to their paper.

The design also adds rank-one smoothing compensation, implemented as another Tensor Core matrix multiply, to recover some accuracy loss. It combines that with asynchronous pipelining, tensor-memory reuse and adaptive partitioning between prefill and decode. These are not free additions. The engineering trade is to add targeted work where it restores quality or overlap, rather than assume the lowest precision is best at every stage.

The Nov. 10, 2026 paper revision reports experiments on an SM100 GPU at 16K–128K contexts for Llama 3.1 8B and Qwen3 14B. The authors report better prefill throughput than selected BF16/FP8 baselines and 2.81 times end-to-end output throughput over a named BF16 FlashAttention baseline. They also report rank-one compensation recovers some measured accuracy loss with about two percent kernel overhead. These are preprint results for a particular setup and benchmark suite, not a claim about all serving stacks or models. (MpFA paper, v2)

The bottleneck moves when the kernel changes

First, matrix multiplication is expensive. Lower precision accelerates it. Then the proportion of time spent in softmax, conversion or memory movement grows. A kernel that looked compute-bound can become limited by an operation that was previously hidden behind multiplication.

That is why a roofline estimate needs complete-kernel measurements. Capture duration, achieved matrix throughput, memory traffic, occupancy and conversion/softmax time where tooling exposes them, plus end-to-end tokens per second and latency under the intended request mix. Nsight Compute and Nsight Systems can help distinguish kernel work from launch gaps and synchronization. NVIDIA’s Blackwell tuning guide details constraints such as occupancy and shared memory.

Check quality alongside speed. Compare attention outputs with a higher-precision reference at representative sequence lengths, then evaluate the full model on relevant tasks. A good average error can still hide long-context failures. A paper’s benchmark set is a starting point, not your acceptance test.

Test more than one batch size and prefill/decode mix. Prefill processes a prompt in parallel and can expose large matrix work; decode advances autoregressively and often has different batch and memory behavior. A kernel tuned to win on long prefill may not improve interactive token latency at low concurrency. Report time-to-first-token, inter-token latency and aggregate throughput separately, and include compilation and warmup policy in repeatable measurements.

Keep the comparison fair: pin the same model weights, attention mask, context, output length and serving scheduler. A benchmark that changes batch size or drops long prompts can improve apparent throughput while changing the user-visible latency distribution. For accuracy, compare both raw attention outputs and downstream model responses; numerical closeness at one layer does not guarantee identical generations. Publish the command, GPU SKU and software versions with the result so another team can tell which part of the design it is reproducing.

Measure the pipeline your service actually runs

Blackwell’s low-precision throughput is useful only when software keeps that compute occupied and avoids wasting cycles around the dataflow. MpFA does not diminish FP4; it shows that kernel engineering decides whether the hardware capability survives contact with real operations. Mixed precision can be a better choice than maximal precision reduction.

For a serving team, the question is not “Can we enable FP4?” It is “Which stage can use FP4 without creating a more expensive conversion, memory or accuracy problem elsewhere?” Benchmark at actual context lengths and concurrency. Pin exact kernels and model revisions, and report prefill and decode separately. The NVFP4 training recipe addresses numerical stability during learning; this paper studies a distinct inference-kernel pipeline. Both show that precision is a system of choices and the bottleneck moves when one changes.