Identifying Sensitive Weights via Post-quantization Integral
Serving Large Language Models (LLMs) is costly. However, post-training weight
quantization can address this problem by both compressing their sizes for
limited memory and saving bandwidth for acceleration. As not all weight
dimensions are equally important, those methods typically rely on a sensitivity
metric, which indicates the element-wise influence of weights on loss function
and is used to preprocess original weights for better quantization. In this
work, we conduct an empirical study on the accuracy of the sensitivity metric,
and find that existing gradient and Hessian based metrics are very inaccurate:
they underestimate quantization's impact on the loss function by orders of
magnitude, mainly due to the small convergence radius of local 2nd order
approximation, \ie, gradient and Hessian term in Taylor's formula. To tackle
this problem, we propose Post-quantization Integral (PQI), an accurate metric
to estimate posterior sensitivity in a fine-grained manner. To leverage this
accurate metric, we further propose ReQuant, a simple yet powerful framework
that mainly consists of two Dense-and-Sparse detach components: self-adaptive
outlier selection and step-wise significant weights detach. Results show that
ReQuant boosts state-of-the-art post-training quantization methods, with a
pronounced improvement of 2.66 perplexity gain on Llama 3.2 1B with QTIP.
67c90398ae4b9276f2d03643
Yuezhou Hu
67c90398ae4b9276f2d03644
Weiyu Huang
67c90398ae4b9276f2d03645
Zichen Liang
admin_assigned
2025-03-07T09:45:50.605Z
67286718746a95c09d04cb1d
/avatars/317efa8459cca08c2ff56c3ab116e15c.svg
Zichen Liang
user
zcliang22
67c90398ae4b9276f2d03646
Chang Chen
67c90398ae4b9276f2d03647
Jintao Zhang
claimed_verified
2025-03-07T09:26:51.606Z
66c0a08bac74db25de8427ec
https://cdn-avatars.huggingface.co/v1/production/uploads/66c0a08bac74db25de8427ec/9D-piDBZqSt6KNkHImmkv.jpeg
Jintao Zhang
user
jt-zhang
67c90398ae4b9276f2d03648
Jun Zhu
67c90398ae4b9276f2d03649
Jianfei Chen
admin_assigned
2025-03-07T09:46:07.369Z
65fcad0ba0d7adc40b54fac2
/avatars/7564b5642378fddb46ec3b5ae57c0402.svg
Jianfei Chen
user
surfingtomchen
Identifying Sensitive Weights via Post-quantization Integral
Serving Large Language Models (LLMs) is costly. However, post-training weight
quantization can address this problem by both compressing their sizes for
limited memory and saving bandwidth for acceleration. As not all weight
dimensions are equally important, those methods typically rely on a sensitivity
metric, which indicates the element-wise influence of weights on loss function
and is used to preprocess original weights for better quantization. In this
work, we conduct an empirical study on the accuracy of the sensitivity metric,
and find that existing gradient and Hessian based metrics are very inaccurate:
they underestimate quantization's impact on the loss function by orders of
magnitude, mainly due to the small convergence radius of local 2nd order
approximation, \ie, gradient and Hessian term in Taylor's formula. To tackle
this problem, we propose Post-quantization Integral (PQI), an accurate metric
to estimate posterior sensitivity in a fine-grained manner. To leverage this
accurate metric, we further propose ReQuant, a simple yet powerful framework
that mainly consists of two Dense-and-Sparse detach components: self-adaptive
outlier selection and step-wise significant weights detach. Results show that
ReQuant boosts state-of-the-art post-training quantization methods, with a
pronounced improvement of 2.66 perplexity gain on Llama 3.2 1B with QTIP.
Post-quantization Integral (PQI) and ReQuant framework improve the accuracy of sensitivity metrics and enhance post-training quantization methods for large language models.
post-training weight quantization sensitivity metric gradient Hessian Taylor's formula Post-quantization Integral PQI ReQuant Dense-and-Sparse detach self-adaptive outlier selection step-wise significant weights detach perplexity gain