VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large
  Language Models
Scaling model size significantly challenges the deployment and inference of
Large Language Models (LLMs). Due to the redundancy in LLM weights, recent
research has focused on pushing weight-only quantization to extremely low-bit
(even down to 2 bits). It reduces memory requirements, optimizes storage costs,
and decreases memory bandwidth needs during inference. However, due to
numerical representation limitations, traditional scalar-based weight
quantization struggles to achieve such extreme low-bit. Recent research on
Vector Quantization (VQ) for LLMs has demonstrated the potential for extremely
low-bit model quantization by compressing vectors into indices using lookup
tables.
  In this paper, we introduce Vector Post-Training Quantization (VPTQ) for
extremely low-bit quantization of LLMs. We use Second-Order Optimization to
formulate the LLM VQ problem and guide our quantization algorithm design by
solving the optimization. We further refine the weights using
Channel-Independent Second-Order Optimization for a granular VQ. In addition,
by decomposing the optimization problem, we propose a brief and effective
codebook initialization algorithm. We also extend VPTQ to support residual and
outlier quantization, which enhances model accuracy and further compresses the
model. Our experimental results show that VPTQ reduces model quantization
perplexity by 0.01-0.34 on LLaMA-2, 0.38-0.68 on Mistral-7B,
4.41-7.34 on LLaMA-3 over SOTA at 2-bit, with an average accuracy
improvement of 0.79-1.5% on LLaMA-2, 1% on Mistral-7B, 11-22% on
LLaMA-3 on QA tasks on average. We only utilize 10.4-18.6% of the
quantization algorithm execution time, resulting in a 1.6-1.8times
increase in inference throughput compared to SOTA.
66f4d188ee391e4c1f788568
Yifei Liu
claimed_verified
2024-10-08T09:05:32.209Z
64df58f8228f31beb8876de4
/avatars/fc08800209a3142e0175d3076f290c3a.svg
Yifei Liu
user
YifeiL
66f4d188ee391e4c1f788569
Jicheng Wen
admin_assigned
2024-09-30T12:31:55.115Z
66e59c49908789cd77addc09
/avatars/9d4753f61e1d654c7fe3e78778daa8cb.svg
GGG
user
JiChengWen
66f4d188ee391e4c1f78856a
Yang Wang
claimed_verified
2024-09-26T08:22:52.135Z
604714a0c82d59b7347b55ae
https://cdn-avatars.huggingface.co/v1/production/uploads/604714a0c82d59b7347b55ae/WZFKDDUPi8JS0BK8t7mIv.jpeg
YangWang92
user
yangwang92
66f4d188ee391e4c1f78856b
Shengyu Ye
admin_assigned
2024-09-30T12:32:01.442Z
6508091067ab943749f9e869
https://cdn-avatars.huggingface.co/v1/production/uploads/6508091067ab943749f9e869/TpdHfIez3rj5wQqrruA7G.jpeg
Shengyu Ye
user
Eviloder
66f4d188ee391e4c1f78856c
Li Lyna Zhang
admin_assigned
2024-09-30T12:32:07.360Z
62b0009c72043b05d29492b2
https://cdn-avatars.huggingface.co/v1/production/uploads/62b0009c72043b05d29492b2/NqRkX2YLhlfOLvYysa7dD.png
Li Lyna Zhang
user
lynazhang
66f4d188ee391e4c1f78856d
Ting Cao
admin_assigned
2024-09-30T12:32:26.099Z
662852bf8bfc90408a97acdf
/avatars/008e6faf51e1be4f0c0d17008dfe2f49.svg
cao
user
tingcao
66f4d188ee391e4c1f78856e
Cheng Li
66f4d188ee391e4c1f78856f
Mao Yang
VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large
  Language Models
Scaling model size significantly challenges the deployment and inference of
Large Language Models (LLMs). Due to the redundancy in LLM weights, recent
research has focused on pushing weight-only quantization to extremely low-bit
(even down to 2 bits). It reduces memory requirements, optimizes storage costs,
and decreases memory bandwidth needs during inference. However, due to
numerical representation limitations, traditional scalar-based weight
quantization struggles to achieve such extreme low-bit. Recent research on
Vector Quantization (VQ) for LLMs has demonstrated the potential for extremely
low-bit model quantization by compressing vectors into indices using lookup
tables.
  In this paper, we introduce Vector Post-Training Quantization (VPTQ) for
extremely low-bit quantization of LLMs. We use Second-Order Optimization to
formulate the LLM VQ problem and guide our quantization algorithm design by
solving the optimization. We further refine the weights using
Channel-Independent Second-Order Optimization for a granular VQ. In addition,
by decomposing the optimization problem, we propose a brief and effective
codebook initialization algorithm. We also extend VPTQ to support residual and
outlier quantization, which enhances model accuracy and further compresses the
model. Our experimental results show that VPTQ reduces model quantization
perplexity by 0.01-0.34 on LLaMA-2, 0.38-0.68 on Mistral-7B,
4.41-7.34 on LLaMA-3 over SOTA at 2-bit, with an average accuracy
improvement of 0.79-1.5% on LLaMA-2, 1% on Mistral-7B, 11-22% on
LLaMA-3 on QA tasks on average. We only utilize 10.4-18.6% of the
quantization algorithm execution time, resulting in a 1.6-1.8times
increase in inference throughput compared to SOTA.
Vector Post-Training Quantization (VPTQ) uses Second-Order Optimization to achieve extremely low-bit quantization of Large Language Models (LLMs), improving accuracy and inference throughput.
Vector Quantization (VQ) Second-Order Optimization Channel-Independent Second-Order Optimization codebook initialization residual quantization outlier quantization perplexity inference throughput