EfficientQAT: Efficient Quantization-Aware Training for Large Language
  Models
Large language models (LLMs) are integral to modern natural language
processing and artificial intelligence. However, they face challenges in
managing their significant memory requirements. Although quantization-aware
training (QAT) offers a solution by reducing memory consumption through low-bit
representations with minimal accuracy loss, it demands substantial training
resources to optimize model weights and quantization parameters. To address
this, we propose Efficient Quantization-Aware Training (EfficientQAT), a novel
quantization technique for compressing LLMs. EfficientQAT involves two
consecutive phases: Block-wise training of all parameters (Block-AP) and
end-to-end training of quantization parameters (E2E-QP). Block-AP sequentially
conducts quantization-aware training for all parameters in each transformer
block with block-wise reconstruction, maintaining efficiency by avoiding
training the entire LLM. Initialized with quantized model, E2E-QP then trains
only quantization parameters (step sizes) end-to-end, enhancing efficiency with
a fixed quantized backbone and reduced trainable parameter count. Extensive
experiments demonstrate that EfficientQAT outperforms previous quantization
methods across a range of models, including base LLMs, instruction-tuned LLMs,
and multimodal LLMs, with scales from 7B to 70B parameters at various
quantization bits. For instance, EfficientQAT obtains a 2-bit Llama-2-70B model
on a single A100-80GB GPU in 41 hours, with less than 3\% accuracy degradation
compared to the full precision (69.48 vs. 72.41). Notably, this INT2 quantized
70B model obtains a 1.67 accuracy gain over the Llama-2-13B model (69.48 vs.
67.81) while requiring less memory (19.2GB vs. 24.2GB). Code is available at
https://github.com/OpenGVLab/EfficientQAT.
6698541ca1c67d388b27285c
Mengzhao Chen
claimed_verified
2024-07-18T09:05:32.220Z
64aea082704210bf815e7551
/avatars/5c8dc0df57596c526b2bccea21835f53.svg
Mengzhao Chen
user
ChenMnZ
6698541ca1c67d388b27285d
Wenqi Shao
6698541ca1c67d388b27285e
Peng Xu
6698541ca1c67d388b27285f
Jiahao Wang
6698541ca1c67d388b272860
Peng Gao
6698541ca1c67d388b272861
Kaipeng Zhang
claimed_verified
2025-06-02T07:57:53.741Z
65f1713552c38a91e0a445e8
/avatars/47ab3ada51c9b9976ac1cd0c4301c373.svg
kaipeng
user
kpzhang996
6698541ca1c67d388b272862
Yu Qiao
6698541ca1c67d388b272863
Ping Luo
EfficientQAT: Efficient Quantization-Aware Training for Large Language
  Models
Large language models (LLMs) are integral to modern natural language
processing and artificial intelligence. However, they face challenges in
managing their significant memory requirements. Although quantization-aware
training (QAT) offers a solution by reducing memory consumption through low-bit
representations with minimal accuracy loss, it demands substantial training
resources to optimize model weights and quantization parameters. To address
this, we propose Efficient Quantization-Aware Training (EfficientQAT), a novel
quantization technique for compressing LLMs. EfficientQAT involves two
consecutive phases: Block-wise training of all parameters (Block-AP) and
end-to-end training of quantization parameters (E2E-QP). Block-AP sequentially
conducts quantization-aware training for all parameters in each transformer
block with block-wise reconstruction, maintaining efficiency by avoiding
training the entire LLM. Initialized with quantized model, E2E-QP then trains
only quantization parameters (step sizes) end-to-end, enhancing efficiency with
a fixed quantized backbone and reduced trainable parameter count. Extensive
experiments demonstrate that EfficientQAT outperforms previous quantization
methods across a range of models, including base LLMs, instruction-tuned LLMs,
and multimodal LLMs, with scales from 7B to 70B parameters at various
quantization bits. For instance, EfficientQAT obtains a 2-bit Llama-2-70B model
on a single A100-80GB GPU in 41 hours, with less than 3\% accuracy degradation
compared to the full precision (69.48 vs. 72.41). Notably, this INT2 quantized
70B model obtains a 1.67 accuracy gain over the Llama-2-13B model (69.48 vs.
67.81) while requiring less memory (19.2GB vs. 24.2GB). Code is available at
https://github.com/OpenGVLab/EfficientQAT.
Efficient quantization-aware training compresses large language models with reduced memory and minimal accuracy loss through block-wise parameter training and end-to-end quantization parameter optimization.
quantization-aware training EfficientQAT Block-wise training end-to-end training quantization parameters transformer block quantized model accuracy degradation INT2 quantization