Quartet: Native FP4 Training Can Be Optimal for Large Language Models
The rapid advancement of large language models (LLMs) has been paralleled by
unprecedented increases in computational demands, with training costs for
state-of-the-art models doubling every few months. Training models directly in
low-precision arithmetic offers a solution, by improving both computational
throughput and energy efficiency. Specifically, NVIDIA's recent Blackwell
architecture facilitates extremely low-precision operations, specifically FP4
variants, promising substantial efficiency gains. Yet, current algorithms for
training LLMs in FP4 precision face significant accuracy degradation and often
rely on mixed-precision fallbacks. In this paper, we systematically investigate
hardware-supported FP4 training and introduce Quartet, a new approach enabling
accurate, end-to-end FP4 training with all the major computations (in e.g.
linear layers) being performed in low precision. Through extensive evaluations
on Llama-type models, we reveal a new low-precision scaling law that quantifies
performance trade-offs across varying bit-widths and allows us to identify a
"near-optimal" low-precision training technique in terms of
accuracy-vs-computation, called Quartet. We implement Quartet using optimized
CUDA kernels tailored for NVIDIA Blackwell GPUs, and show that it can achieve
state-of-the-art accuracy for FP4 precision, successfully training
billion-scale models. Our method demonstrates that fully FP4-based training is
a competitive alternative to standard-precision and FP8 training. Our code is
available at https://github.com/IST-DASLab/Quartet.
682da9d3781210358218a950
Roberto L. Castro
682da9d3781210358218a951
Andrei Panferov
claimed_verified
2025-05-26T14:07:46.802Z
623753b5eddd7763adc9346a
https://cdn-avatars.huggingface.co/v1/production/uploads/623753b5eddd7763adc9346a/rcpQAKZNrkn1-tMtraQBX.jpeg
Andrei Panferov
user
BlackSamorez
682da9d3781210358218a952
Soroush Tabesh
admin_assigned
2025-05-26T14:25:12.072Z
632a2e325f2ff1958c0103be
https://cdn-avatars.huggingface.co/v1/production/uploads/632a2e325f2ff1958c0103be/Tb0ql9e4LcaFktTK1hzqe.jpeg
Rush Tabesh
user
soroushtabesh
682da9d3781210358218a953
Oliver Sieberling
admin_assigned
2025-05-26T14:25:18.736Z
659ddfb45673a33b5db22d57
/avatars/ae1dce603b4cae2659d6070e8ce98b15.svg
Oliver Sieberling
user
OliverSieberling
682da9d3781210358218a954
Jiale Chen
claimed_verified
2025-07-29T07:11:23.593Z
6298d8dab58e71e2ac9e2967
/avatars/99fec4ba78ab61c5952f51e6ebf03ffa.svg
Jiale Chen
user
softmax
682da9d3781210358218a955
Mahdi Nikdan
admin_assigned
2025-05-26T14:25:45.476Z
6526b8ebba9a8279c139616b
/avatars/09f6b677603a03be128996a0765233e6.svg
Mahdi Nikdan
user
mnikdan97
682da9d3781210358218a956
Saleh Ashkboos
682da9d3781210358218a957
Dan Alistarh
admin_assigned
2025-05-26T14:25:55.295Z
64d100c5d8d0927372e3d4c0
/avatars/91d9e4f1dab25b70d901783cdfcd2fd1.svg
Dan Alistarh
user
dalistarh
Quartet: Native FP4 Training Can Be Optimal for Large Language Models
The rapid advancement of large language models (LLMs) has been paralleled by
unprecedented increases in computational demands, with training costs for
state-of-the-art models doubling every few months. Training models directly in
low-precision arithmetic offers a solution, by improving both computational
throughput and energy efficiency. Specifically, NVIDIA's recent Blackwell
architecture facilitates extremely low-precision operations, specifically FP4
variants, promising substantial efficiency gains. Yet, current algorithms for
training LLMs in FP4 precision face significant accuracy degradation and often
rely on mixed-precision fallbacks. In this paper, we systematically investigate
hardware-supported FP4 training and introduce Quartet, a new approach enabling
accurate, end-to-end FP4 training with all the major computations (in e.g.
linear layers) being performed in low precision. Through extensive evaluations
on Llama-type models, we reveal a new low-precision scaling law that quantifies
performance trade-offs across varying bit-widths and allows us to identify a
"near-optimal" low-precision training technique in terms of
accuracy-vs-computation, called Quartet. We implement Quartet using optimized
CUDA kernels tailored for NVIDIA Blackwell GPUs, and show that it can achieve
state-of-the-art accuracy for FP4 precision, successfully training
billion-scale models. Our method demonstrates that fully FP4-based training is
a competitive alternative to standard-precision and FP8 training. Our code is
available at https://github.com/IST-DASLab/Quartet.
Quartet, a hardware-supported FP4 training approach for large language models, demonstrates state-of-the-art accuracy while significantly reducing computational costs compared to standard or FP8 precision.
large language models low-precision arithmetic Blackwell architecture FP4 mixed-precision linear layers low-precision scaling law CUDA kernels