Pretraining Large Language Models with NVFP4
Large Language Models (LLMs) today are powerful problem solvers across many
domains, and they continue to get stronger as they scale in model size,
training set size, and training set quality, as shown by extensive research and
experimentation across the industry. Training a frontier model today requires
on the order of tens to hundreds of yottaflops, which is a massive investment
of time, compute, and energy. Improving pretraining efficiency is therefore
essential to enable the next generation of even more capable LLMs. While 8-bit
floating point (FP8) training is now widely adopted, transitioning to even
narrower precision, such as 4-bit floating point (FP4), could unlock additional
improvements in computational speed and resource utilization. However,
quantization at this level poses challenges to training stability, convergence,
and implementation, notably for large-scale models trained on long token
horizons.
  In this study, we introduce a novel approach for stable and accurate training
of large language models (LLMs) using the NVFP4 format. Our method integrates
Random Hadamard transforms (RHT) to bound block-level outliers, employs a
two-dimensional quantization scheme for consistent representations across both
the forward and backward passes, utilizes stochastic rounding for unbiased
gradient estimation, and incorporates selective high-precision layers. We
validate our approach by training a 12-billion-parameter model on 10 trillion
tokens -- the longest publicly documented training run in 4-bit precision to
date. Our results show that the model trained with our NVFP4-based pretraining
technique achieves training loss and downstream task accuracies comparable to
an FP8 baseline. These findings highlight that NVFP4, when combined with our
training approach, represents a major step forward in narrow-precision LLM
training algorithms.
68db57aed2bf1f4b15ec7538
NVIDIA
68db57aed2bf1f4b15ec7539
Felix Abecassis
68db57aed2bf1f4b15ec753a
Anjulie Agrusa
68db57aed2bf1f4b15ec753b
Dong Ahn
68db57aed2bf1f4b15ec753c
Jonah Alben
68db57aed2bf1f4b15ec753d
Stefania Alborghetti
68db57aed2bf1f4b15ec753e
Michael Andersch
68db57aed2bf1f4b15ec753f
Sivakumar Arayandi
68db57aed2bf1f4b15ec7540
Alexis Bjorlin
68db57aed2bf1f4b15ec7541
Aaron Blakeman
68db57aed2bf1f4b15ec7542
Evan Briones
68db57aed2bf1f4b15ec7543
Ian Buck
68db57aed2bf1f4b15ec7544
Bryan Catanzaro
68db57aed2bf1f4b15ec7545
Jinhang Choi
68db57aed2bf1f4b15ec7546
Mike Chrzanowski
68db57aed2bf1f4b15ec7547
Eric Chung
68db57aed2bf1f4b15ec7548
Victor Cui
68db57aed2bf1f4b15ec7549
Steve Dai
68db57aed2bf1f4b15ec754a
Bita Darvish Rouhani
68db57aed2bf1f4b15ec754b
Carlo del Mundo
68db57aed2bf1f4b15ec754c
Deena Donia
68db57aed2bf1f4b15ec754d
Burc Eryilmaz
68db57aed2bf1f4b15ec754e
Henry Estela
68db57aed2bf1f4b15ec754f
Abhinav Goel
68db57aed2bf1f4b15ec7550
Oleg Goncharov
68db57aed2bf1f4b15ec7551
Yugi Guvvala
68db57aed2bf1f4b15ec7552
Robert Hesse
68db57aed2bf1f4b15ec7553
Russell Hewett
68db57aed2bf1f4b15ec7554
Herbert Hum
68db57aed2bf1f4b15ec7555
Ujval Kapasi
68db57aed2bf1f4b15ec7556
Brucek Khailany
68db57aed2bf1f4b15ec7557
Mikail Khona
68db57aed2bf1f4b15ec7558
Nick Knight
68db57aed2bf1f4b15ec7559
Alex Kondratenko
68db57aed2bf1f4b15ec755a
Ronny Krashinsky
68db57aed2bf1f4b15ec755b
Ben Lanir
68db57aed2bf1f4b15ec755c
Simon Layton
68db57aed2bf1f4b15ec755d
Michael Lightstone
68db57aed2bf1f4b15ec755e
Daniel Lo
68db57aed2bf1f4b15ec755f
Paulius Micikevicius
68db57aed2bf1f4b15ec7560
Asit Mishra
68db57aed2bf1f4b15ec7561
Tim Moon
68db57aed2bf1f4b15ec7562
Deepak Narayanan
68db57aed2bf1f4b15ec7563
Chao Ni
68db57aed2bf1f4b15ec7564
Abhijit Paithankar
68db57aed2bf1f4b15ec7565
Satish Pasumarthi
68db57aed2bf1f4b15ec7566
Ankit Patel
68db57aed2bf1f4b15ec7567
Mostofa Patwary
68db57aed2bf1f4b15ec7568
Ashwin Poojary
68db57aed2bf1f4b15ec7569
Gargi Prasad
68db57aed2bf1f4b15ec756a
Sweta Priyadarshi
68db57aed2bf1f4b15ec756b
Yigong Qin
68db57aed2bf1f4b15ec756c
Xiaowei Ren
68db57aed2bf1f4b15ec756d
Oleg Rybakov
68db57aed2bf1f4b15ec756e
Charbel Sakr
68db57aed2bf1f4b15ec756f
Sanjeev Satheesh
68db57aed2bf1f4b15ec7570
Stas Sergienko
68db57aed2bf1f4b15ec7571
Pasha Shamis
68db57aed2bf1f4b15ec7572
Kirthi Shankar
claimed_verified
2025-10-01T10:24:40.556Z
660cfd250ebb657f487339ad
/avatars/eb3a68a9bcb68839d6532a92e41028ac.svg
Kirthi Shankar Sivamani
user
ksivaman
68db57aed2bf1f4b15ec7573
Nishant Sharma
68db57aed2bf1f4b15ec7574
Mohammad Shoeybi
68db57aed2bf1f4b15ec7575
Michael Siu
68db57aed2bf1f4b15ec7576
Misha Smelyanskiy
68db57aed2bf1f4b15ec7577
Darko Stosic
68db57aed2bf1f4b15ec7578
Dusan Stosic
68db57aed2bf1f4b15ec7579
Bor-Yiing Su
68db57aed2bf1f4b15ec757a
Frank Sun
68db57aed2bf1f4b15ec757b
Nima Tajbakhsh
68db57aed2bf1f4b15ec757c
Shelby Thomas
68db57aed2bf1f4b15ec757d
Przemek Tredak
68db57aed2bf1f4b15ec757e
Evgeny Tsykunov
68db57aed2bf1f4b15ec757f
Gandhi Vaithilingam
68db57aed2bf1f4b15ec7580
Aditya Vavre
68db57aed2bf1f4b15ec7581
Rangharajan Venkatesan
68db57aed2bf1f4b15ec7582
Roger Waleffe
68db57aed2bf1f4b15ec7583
Qiyu Wan
68db57aed2bf1f4b15ec7584
Hexin Wang
68db57aed2bf1f4b15ec7585
Mengdi Wang
68db57aed2bf1f4b15ec7586
Lizzie Wei
68db57aed2bf1f4b15ec7587
Hao Wu
68db57aed2bf1f4b15ec7588
Evan Wu
68db57aed2bf1f4b15ec7589
Keith Wyss
68db57aed2bf1f4b15ec758a
Ning Xu
68db57aed2bf1f4b15ec758b
Jinze Xue
68db57aed2bf1f4b15ec758c
Charlene Yang
68db57aed2bf1f4b15ec758d
Yujia Zhai
68db57aed2bf1f4b15ec758e
Ruoxi Zhang
68db57aed2bf1f4b15ec758f
Jingyang Zhu
68db57aed2bf1f4b15ec7590
Zhongbo Zhu
Pretraining Large Language Models with NVFP4
Large Language Models (LLMs) today are powerful problem solvers across many
domains, and they continue to get stronger as they scale in model size,
training set size, and training set quality, as shown by extensive research and
experimentation across the industry. Training a frontier model today requires
on the order of tens to hundreds of yottaflops, which is a massive investment
of time, compute, and energy. Improving pretraining efficiency is therefore
essential to enable the next generation of even more capable LLMs. While 8-bit
floating point (FP8) training is now widely adopted, transitioning to even
narrower precision, such as 4-bit floating point (FP4), could unlock additional
improvements in computational speed and resource utilization. However,
quantization at this level poses challenges to training stability, convergence,
and implementation, notably for large-scale models trained on long token
horizons.
  In this study, we introduce a novel approach for stable and accurate training
of large language models (LLMs) using the NVFP4 format. Our method integrates
Random Hadamard transforms (RHT) to bound block-level outliers, employs a
two-dimensional quantization scheme for consistent representations across both
the forward and backward passes, utilizes stochastic rounding for unbiased
gradient estimation, and incorporates selective high-precision layers. We
validate our approach by training a 12-billion-parameter model on 10 trillion
tokens -- the longest publicly documented training run in 4-bit precision to
date. Our results show that the model trained with our NVFP4-based pretraining
technique achieves training loss and downstream task accuracies comparable to
an FP8 baseline. These findings highlight that NVFP4, when combined with our
training approach, represents a major step forward in narrow-precision LLM
training algorithms.
A novel training approach using NVFP4 format with Random Hadamard transforms, two-dimensional quantization, stochastic rounding, and selective high-precision layers enables stable and accurate training of large language models in 4-bit precision.
Large Language Models (LLMs) 8-bit floating point (FP8) 4-bit floating point (FP4) quantization training stability convergence implementation Random Hadamard transforms (RHT) two-dimensional quantization stochastic rounding selective high-precision layers NVFP4 training loss downstream task accuracies