Quartet: Towards Optimal End-to-End FP4 Training for Large Language Models
Roberto Castro, Andrei Panferov, Rush Tabesh, Oliver Sieberling, Jiale Chen, Mahdi Nikdan, Saleh Ashkboos, Dan Alistarh
The rapid advancement of large language models (LLMs) has been paralleled by unprecedented increases in computational demands, with training costs doubling every few months. Training models directly in low-precision arithmetic offers a solution, by improving both computational throughput and energy efficiency. Specifically, NVIDIA's recent Blackwell architecture facilitates extremely low-precision operations, specifically FP4, promising substantial efficiency gains. Yet, current algorithms for training LLMs in FP4 precision face significant accuracy degradation and often rely on mixed-precision fallbacks. In this paper, we systematically investigate hardware-supported FP4 training and introduce Quartet, a new approach enabling accurate, end-to-end FP4 training with all the major computations (in e.g. linear layers) being performed in low precision. Through extensive evaluations on Llama-type models, we reveal a new low-precision scaling law that quantifies performance trade-offs across varying bit-widths. Implementing optimized CUDA kernels tailored for NVIDIA Blackwell GPUs, Quartet achieves state-of-the-art accuracy, successfully training billion-scale models. Our method sets a new benchmark in efficiency-vs-accuracy, demonstrating that fully FP4-based training is a competitive alternative to standard-precision training.