QuEST: Stable Training of LLMs with 1-Bit Weights and Activations
One approach to reducing the massive costs of large language models (LLMs) is
the use of quantized or sparse representations for training or deployment.
While post-training compression methods are very popular, the question of
obtaining even more accurate compressed models by directly training over such
representations, i.e., Quantization-Aware Training (QAT), is still open: for
example, a recent study (arXiv:2411.04330v2) put the "optimal" bit-width at
which models can be trained using QAT, while staying accuracy-competitive with
standard FP16/BF16 precision, at 8-bits weights and activations.
  We advance this state-of-the-art via a new method called QuEST, which is
Pareto-competitive with FP16, i.e., it provides better accuracy at lower model
size, while training models with weights and activations in 4-bits or less.
Moreover, QuEST allows stable training with 1-bit weights and activations.
QuEST achieves this by improving two key aspects of QAT methods: (1) accurate
and fast quantization of the (continuous) distributions of weights and
activations via Hadamard normalization and MSE-optimal fitting; (2) a new trust
gradient estimator based on the idea of explicitly minimizing the error between
the noisy gradient computed over quantized states and the "true" (but unknown)
full-precision gradient. Experiments on Llama-type architectures show that
QuEST induces stable scaling laws across the entire range of hardware-supported
precisions, and can be extended to sparse representations. We provide GPU
kernel support showing that models produced by QuEST can be executed
efficiently. Our code is available at https://github.com/IST-DASLab/QuEST.
67a9b1a69a99341e859c488d
Andrei Panferov
extracted_confirmed
2025-02-10T08:09:18.686Z
623753b5eddd7763adc9346a
https://cdn-avatars.huggingface.co/v1/production/uploads/623753b5eddd7763adc9346a/rcpQAKZNrkn1-tMtraQBX.jpeg
Andrei Panferov
user
BlackSamorez
67a9b1a69a99341e859c488e
Jiale Chen
admin_assigned
2025-07-30T20:09:57.675Z
6298d8dab58e71e2ac9e2967
/avatars/99fec4ba78ab61c5952f51e6ebf03ffa.svg
Jiale Chen
user
softmax
67a9b1a69a99341e859c488f
Soroush Tabesh
claimed_verified
2025-02-10T09:49:37.573Z
632a2e325f2ff1958c0103be
https://cdn-avatars.huggingface.co/v1/production/uploads/632a2e325f2ff1958c0103be/Tb0ql9e4LcaFktTK1hzqe.jpeg
Rush Tabesh
user
soroushtabesh
67a9b1a69a99341e859c4890
Roberto L. Castro
67a9b1a69a99341e859c4891
Mahdi Nikdan
claimed_verified
2025-02-10T09:50:25.944Z
6526b8ebba9a8279c139616b
/avatars/09f6b677603a03be128996a0765233e6.svg
Mahdi Nikdan
user
mnikdan97
67a9b1a69a99341e859c4892
Dan Alistarh
claimed_verified
2025-02-10T09:49:35.449Z
64ef52c2718f94ae8e78a5e7
/avatars/d169f4ee62786a3eb4a3fa9d1fec52e9.svg
Alistarh
user
d-alistarh
QuEST: Stable Training of LLMs with 1-Bit Weights and Activations
One approach to reducing the massive costs of large language models (LLMs) is
the use of quantized or sparse representations for training or deployment.
While post-training compression methods are very popular, the question of
obtaining even more accurate compressed models by directly training over such
representations, i.e., Quantization-Aware Training (QAT), is still open: for
example, a recent study (arXiv:2411.04330v2) put the "optimal" bit-width at
which models can be trained using QAT, while staying accuracy-competitive with
standard FP16/BF16 precision, at 8-bits weights and activations.
  We advance this state-of-the-art via a new method called QuEST, which is
Pareto-competitive with FP16, i.e., it provides better accuracy at lower model
size, while training models with weights and activations in 4-bits or less.
Moreover, QuEST allows stable training with 1-bit weights and activations.
QuEST achieves this by improving two key aspects of QAT methods: (1) accurate
and fast quantization of the (continuous) distributions of weights and
activations via Hadamard normalization and MSE-optimal fitting; (2) a new trust
gradient estimator based on the idea of explicitly minimizing the error between
the noisy gradient computed over quantized states and the "true" (but unknown)
full-precision gradient. Experiments on Llama-type architectures show that
QuEST induces stable scaling laws across the entire range of hardware-supported
precisions, and can be extended to sparse representations. We provide GPU
kernel support showing that models produced by QuEST can be executed
efficiently. Our code is available at https://github.com/IST-DASLab/QuEST.
QuEST, a new method for quantization-aware training, achieves better accuracy at lower model size with weights and activations in 4-bits or less, even stable training with 1-bit, by improving quantization accuracy and the gradient estimator.
Quantization-Aware Training (QAT) QuEST Hadamard normalization MSE-optimal fitting trust gradient estimator Llama-type architectures scaling laws sparse representations