NVFP4 stability and performance: what academic studies and benchmarks say

In the fast-moving world of artificial intelligence, it is not only the models that make headlines. The numeric formats used to represent calculations play a crucial role. In 2025, NVIDIA released its NVFP4, a 4-bit quantization format designed to train or run LLMs with fewer resources while keeping accuracy intact. To better understand NVFP4, you can check our dedicated page.

For developers and enterprises, it is hard to navigate through the endless AI announcements: is it worth it? To answer that, we dive into academic research, independent benchmarks, and already deployed use cases.

---

Why stability matters with NVFP4

Moving from 16-bit (BF16) to 8-bit (FP8) was already a breakthrough, yet in many domains BF16 or FP16 are still preferred to minimize accuracy loss. Dropping down to 4-bit like with NVFP4 means working with a razor-thin margin for error.

• Too much rounding, and the model becomes unstable.
• Too much precision loss, and performance collapses.

To avoid these pitfalls, the quantization method must be extremely precise. As with other advanced quantization approaches, it is not about converting everything into 4-bit.

---

What academic research says

The strongest validation comes from a scientific paper published on arXiv by Abecassis et al. in September 2025.

• Researchers trained a hybrid Mamba-Transformer model with 12 billion parameters on 10 trillion tokens, the largest FP4 training ever published.
• Results showed the NVFP4 loss curve closely tracked FP8, with only 1 to 1.5% gap.
• On final benchmarks (reasoning, mathematics, multilingual tasks), NVFP4 matched FP8, with only a minor drop on coding tasks, attributed to dataset noise rather than the format itself.

---

Techniques that ensure NVFP4 stability

If NVFP4 works, it is not by chance. Researchers combine several techniques:

• Selective high-precision layers: only the most sensitive final layers remain in FP8 or BF16.
• Random Hadamard transforms: a mathematical method to reduce variance.
• 2D quantization: for better value distribution handling.
• Stochastic rounding: prevents bias from systematic rounding.

These elements were confirmed in the arXiv study. Removing just one of them caused clear degradation in convergence.

---

Real-world cases: DeepSeek-R1 and Nemotron

• DeepSeek-R1: this model tested NVFP4 in production, showing notable gains in inference throughput and energy efficiency, as reported by Introl Tech Blog.
• Nemotron: NVIDIA’s open source initiative recommends NVFP4 as the preferred 4-bit training format and even provides GitHub examples for easy integration (NVIDIA blog).

These examples show NVFP4 is no longer just an academic curiosity, but is progressively entering the production ecosystem.

---

Limitations and unanswered questions

As promising as it is, NVFP4 still raises questions:

• Hardware compatibility: mainly optimized for Blackwell GPUs. Previous generations (Ampere, Hopper) do not fully benefit from it.
• Adoption rate: so far adoption is progressive. NVIDIA’s tools (Transformer Engine, TensorRT-LLM) support it, as well as third-party solutions like vLLM and Hugging Face Transformers. Many models are already available in NVFP4 on Hugging Face.
• Task diversity: most validations focus on LLM pretraining.
• To verify: how does it perform in vision, multimodal tasks, or complex generative use cases?

---

For developers and enterprises: a format to watch closely

For developers:

• NVFP4 enables training locally or running inference on larger models with less VRAM.
• It helps reduce energy costs of a cluster.
• It accelerates rapid prototyping.

For enterprises:

• NVFP4 can cut cloud costs by using 4-bit models without major performance loss.
• It helps increase server capacity, allowing more models to be hosted on the same infrastructure.

It is similar to moving from a classic warehouse to an optimized storage system: same surface area, but twice as many products fit inside.

---

Conclusion: stability proven, adoption in progress

NVFP4 ticks many boxes:

• validated academically (12B / 10T tokens with proven stability),
• confirmed by industrial benchmarks (Tom’s Hardware, NVIDIA),
• already adopted in real-world projects (DeepSeek-R1, Nemotron).

The question is whether adoption will go beyond NVIDIA’s ecosystem and become an industry standard, as FP8 did before. For both enterprises and developers, one thing is certain: in 2025, evaluating NVFP4 for your projects is essential.

---

NVFP4 article series

1. NVFP4: understanding NVIDIA’s new 4-bit format for AI
2. NVFP4 vs FP8 vs BF16 vs MXFP4: comparing low-precision formats for AI
3. NVFP4 stability and performance: what academic studies and benchmarks say ← You are here
4. Why NVFP4 matters for business: costs, speed and adoption in AI

---