Scaling Law for Quantization-Aware Training
Large language models (LLMs) demand substantial computational and memory
resources, creating deployment challenges. Quantization-aware training (QAT)
addresses these challenges by reducing model precision while maintaining
performance. However, the scaling behavior of QAT, especially at 4-bit
precision (W4A4), is not well understood. Existing QAT scaling laws often
ignore key factors such as the number of training tokens and quantization
granularity, which limits their applicability. This paper proposes a unified
scaling law for QAT that models quantization error as a function of model size,
training data volume, and quantization group size. Through 268 QAT experiments,
we show that quantization error decreases as model size increases, but rises
with more training tokens and coarser quantization granularity. To identify the
sources of W4A4 quantization error, we decompose it into weight and activation
components. Both components follow the overall trend of W4A4 quantization
error, but with different sensitivities. Specifically, weight quantization
error increases more rapidly with more training tokens. Further analysis shows
that the activation quantization error in the FC2 layer, caused by outliers, is
the primary bottleneck of W4A4 QAT quantization error. By applying
mixed-precision quantization to address this bottleneck, we demonstrate that
weight and activation quantization errors can converge to similar levels.
Additionally, with more training data, weight quantization error eventually
exceeds activation quantization error, suggesting that reducing weight
quantization error is also important in such scenarios. These findings offer
key insights for improving QAT research and development.
682e887e866a44f6a81409b1
Mengzhao Chen
claimed_verified
2025-05-22T07:16:29.873Z
64aea082704210bf815e7551
/avatars/5c8dc0df57596c526b2bccea21835f53.svg
Mengzhao Chen
user
ChenMnZ
682e887e866a44f6a81409b2
Chaoyi Zhang
682e887e866a44f6a81409b3
Jing Liu
682e887e866a44f6a81409b4
Yutao Zeng
claimed_verified
2025-05-26T08:22:15.288Z
6371128eafbe42caa5a5222b
/avatars/c3b2ab35949c38aa3dfb2657a1300aac.svg
Yutao Zeng
user
Taoer
682e887e866a44f6a81409b5
Zeyue Xue
682e887e866a44f6a81409b6
Zhiheng Liu
682e887e866a44f6a81409b7
Yunshui Li
claimed_verified
2025-06-19T15:11:33.572Z
62e670d33651180f7d334ef3
https://cdn-avatars.huggingface.co/v1/production/uploads/1661829952739-62e670d33651180f7d334ef3.jpeg
Li Yunshui
user
Wa2erGo
682e887e866a44f6a81409b8
Jin Ma
682e887e866a44f6a81409b9
Jie Huang
682e887e866a44f6a81409ba
Xun Zhou
682e887e866a44f6a81409bb
Ping Luo
Scaling Law for Quantization-Aware Training
Large language models (LLMs) demand substantial computational and memory
resources, creating deployment challenges. Quantization-aware training (QAT)
addresses these challenges by reducing model precision while maintaining
performance. However, the scaling behavior of QAT, especially at 4-bit
precision (W4A4), is not well understood. Existing QAT scaling laws often
ignore key factors such as the number of training tokens and quantization
granularity, which limits their applicability. This paper proposes a unified
scaling law for QAT that models quantization error as a function of model size,
training data volume, and quantization group size. Through 268 QAT experiments,
we show that quantization error decreases as model size increases, but rises
with more training tokens and coarser quantization granularity. To identify the
sources of W4A4 quantization error, we decompose it into weight and activation
components. Both components follow the overall trend of W4A4 quantization
error, but with different sensitivities. Specifically, weight quantization
error increases more rapidly with more training tokens. Further analysis shows
that the activation quantization error in the FC2 layer, caused by outliers, is
the primary bottleneck of W4A4 QAT quantization error. By applying
mixed-precision quantization to address this bottleneck, we demonstrate that
weight and activation quantization errors can converge to similar levels.
Additionally, with more training data, weight quantization error eventually
exceeds activation quantization error, suggesting that reducing weight
quantization error is also important in such scenarios. These findings offer
key insights for improving QAT research and development.
A unified scaling law for quantization-aware training (QAT) identifies key factors affecting quantization error, leading to improvements through mixed-precision quantization.
quantization-aware training QAT quantization error model size training tokens quantization granularity weight quantization activation quantization mixed-precision quantization FC2 layer