QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models
Recently years have witnessed a rapid development of large language models
(LLMs). Despite the strong ability in many language-understanding tasks, the
heavy computational burden largely restricts the application of LLMs especially
when one needs to deploy them onto edge devices. In this paper, we propose a
quantization-aware low-rank adaptation (QA-LoRA) algorithm. The motivation lies
in the imbalanced degrees of freedom of quantization and adaptation, and the
solution is to use group-wise operators which increase the degree of freedom of
quantization meanwhile decreasing that of adaptation. QA-LoRA is easily
implemented with a few lines of code, and it equips the original LoRA with
two-fold abilities: (i) during fine-tuning, the LLM's weights are quantized
(e.g., into INT4) to reduce time and memory usage; (ii) after fine-tuning, the
LLM and auxiliary weights are naturally integrated into a quantized model
without loss of accuracy. We apply QA-LoRA to the LLaMA and LLaMA2 model
families and validate its effectiveness in different fine-tuning datasets and
downstream scenarios. Code will be made available at
https://github.com/yuhuixu1993/qa-lora.
65139696ddff6f95293dcf18
Yuhui Xu
claimed_verified
2025-02-03T11:11:43.559Z
6602869253a0518b2a98cafd
/avatars/c14b5953a716f42c83ad28147f8308ae.svg
Yuhui Xu
user
yuhuixu
65139696ddff6f95293dcf19
Lingxi Xie
65139696ddff6f95293dcf1a
Xiaotao Gu
claimed_verified
2025-07-02T09:25:23.064Z
6438bca4a5e10f6d58694b47
/avatars/3aeb25fbc73c5cab1265e13d11adfb76.svg
Xiaotao Gu
user
xgeric
65139696ddff6f95293dcf1b
Xin Chen
65139696ddff6f95293dcf1c
Heng Chang
claimed_verified
2023-09-28T08:20:19.783Z
64c3698f007b906a75a14943
/avatars/da31d842225cc6074ecd904a82caf931.svg
Heng Chang
user
hchang95
65139696ddff6f95293dcf1d
Hengheng Zhang
admin_assigned
2023-09-27T10:11:54.786Z
63be2d5fb3b8c44f8ced02eb
/avatars/dc2eb7e058e48a203155166a0a967900.svg
Hengheng Zhang
user
imhmhm
65139696ddff6f95293dcf1e
Zhensu Chen
65139696ddff6f95293dcf1f
Xiaopeng Zhang
admin_assigned
2023-09-27T10:12:02.549Z
640bea9cb4f73086cddbe0a6
/avatars/ae460dd971de1d04660b266b22f81f9d.svg
Xiaopeng Zhang
user
xpzhang
65139696ddff6f95293dcf20
Qi Tian
QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models
Recently years have witnessed a rapid development of large language models
(LLMs). Despite the strong ability in many language-understanding tasks, the
heavy computational burden largely restricts the application of LLMs especially
when one needs to deploy them onto edge devices. In this paper, we propose a
quantization-aware low-rank adaptation (QA-LoRA) algorithm. The motivation lies
in the imbalanced degrees of freedom of quantization and adaptation, and the
solution is to use group-wise operators which increase the degree of freedom of
quantization meanwhile decreasing that of adaptation. QA-LoRA is easily
implemented with a few lines of code, and it equips the original LoRA with
two-fold abilities: (i) during fine-tuning, the LLM's weights are quantized
(e.g., into INT4) to reduce time and memory usage; (ii) after fine-tuning, the
LLM and auxiliary weights are naturally integrated into a quantized model
without loss of accuracy. We apply QA-LoRA to the LLaMA and LLaMA2 model
families and validate its effectiveness in different fine-tuning datasets and
downstream scenarios. Code will be made available at
https://github.com/yuhuixu1993/qa-lora.
QA-LoRA algorithm reduces computational requirements for large language models by quantizing weights during fine-tuning and integrating auxiliary weights post-training.
quantization-aware low-rank adaptation QA-LoRA group-wise operators quantization fine-tuning LLaMA LLaMA2