LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models
Quantization is an indispensable technique for serving Large Language Models
(LLMs) and has recently found its way into LoRA fine-tuning. In this work we
focus on the scenario where quantization and LoRA fine-tuning are applied
together on a pre-trained model. In such cases it is common to observe a
consistent gap in the performance on downstream tasks between full fine-tuning
and quantization plus LoRA fine-tuning approach. In response, we propose LoftQ
(LoRA-Fine-Tuning-aware Quantization), a novel quantization framework that
simultaneously quantizes an LLM and finds a proper low-rank initialization for
LoRA fine-tuning. Such an initialization alleviates the discrepancy between the
quantized and full-precision model and significantly improves the
generalization in downstream tasks. We evaluate our method on natural language
understanding, question answering, summarization, and natural language
generation tasks. Experiments show that our method is highly effective and
outperforms existing quantization methods, especially in the challenging 2-bit
and 2/4-bit mixed precision regimes. We will release our code.
652c9bda6cdea405851d35eb
Yixiao Li
claimed_verified
2023-10-16T15:01:44.164Z
651da7b2c34673d710edd9e4
https://cdn-avatars.huggingface.co/v1/production/uploads/651da7b2c34673d710edd9e4/I8SylBR4sL8XLsGc2LlQA.png
LoftQ
user
LoftQ
652c9bda6cdea405851d35ec
Yifan Yu
claimed_verified
2023-10-19T08:10:29.059Z
63d9e54d25575d85ad0b90c8
/avatars/c936aba537e3336057157582a8e843f9.svg
Yifan
user
YYF42
652c9bda6cdea405851d35ed
Chen Liang
claimed_verified
2023-10-17T07:31:08.018Z
63e6b5e22d2c508de9001afd
/avatars/43cec9e8b8d490bd259e383954846a1e.svg
Chen Liang
user
cliang1453
652c9bda6cdea405851d35ee
Pengcheng He
admin_assigned
2023-10-16T08:17:50.286Z
5efd09cf49ed724c8a135868
/avatars/af12bc94657979677a9f26183f0c9727.svg
Pengcheng He
user
DeBERTa
652c9bda6cdea405851d35ef
Nikos Karampatziakis
admin_assigned
2023-10-16T08:17:58.101Z
64d6785b089bc502cea059be
/avatars/79a504d2b66aa256327c8d11243d41b0.svg
Nikos Karampatziakis
user
n17s
652c9bda6cdea405851d35f0
Weizhu Chen
admin_assigned
2023-10-16T08:18:05.198Z
64da876370446182be5b608d
/avatars/e412fdc71404ecdf638e416846e3ebfb.svg
Weizhu Chen
user
chenweizhu
652c9bda6cdea405851d35f1
Tuo Zhao
extracted_pending
2025-04-07T19:26:13.507Z
67f426c9cf0cf195e2220600
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Sz8qQ1ltAi2V33jPvycY4.png
Tuo Zhao
user
tourzhao
LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models
Quantization is an indispensable technique for serving Large Language Models
(LLMs) and has recently found its way into LoRA fine-tuning. In this work we
focus on the scenario where quantization and LoRA fine-tuning are applied
together on a pre-trained model. In such cases it is common to observe a
consistent gap in the performance on downstream tasks between full fine-tuning
and quantization plus LoRA fine-tuning approach. In response, we propose LoftQ
(LoRA-Fine-Tuning-aware Quantization), a novel quantization framework that
simultaneously quantizes an LLM and finds a proper low-rank initialization for
LoRA fine-tuning. Such an initialization alleviates the discrepancy between the
quantized and full-precision model and significantly improves the
generalization in downstream tasks. We evaluate our method on natural language
understanding, question answering, summarization, and natural language
generation tasks. Experiments show that our method is highly effective and
outperforms existing quantization methods, especially in the challenging 2-bit
and 2/4-bit mixed precision regimes. We will release our code.
LoftQ, a novel quantization framework, improves performance of LLMs by simultaneously quantizing and initializing LoRA fine-tuning, reducing the gap in downstream task performance.
quantization Large Language Models (LLMs) LoRA fine-tuning LoftQ low-rank initialization generalization natural language understanding question answering summarization natural language generation 2-bit 2/4-bit mixed precision