MultiLoRA: Democratizing LoRA for Better Multi-Task Learning
LoRA achieves remarkable resource efficiency and comparable performance when
adapting LLMs for specific tasks. Since ChatGPT demonstrated superior
performance on various tasks, there has been a growing desire to adapt one
model for all tasks. However, the explicit low-rank of LoRA limits the
adaptation performance in complex multi-task scenarios. LoRA is dominated by a
small number of top singular vectors while fine-tuning decomposes into a set of
less important unitary transforms. In this paper, we propose MultiLoRA for
better multi-task adaptation by reducing the dominance of top singular vectors
observed in LoRA. MultiLoRA scales LoRA modules horizontally and change
parameter initialization of adaptation matrices to reduce parameter dependency,
thus yields more balanced unitary subspaces. We unprecedentedly construct
specialized training data by mixing datasets of instruction follow, natural
language understanding, world knowledge, to cover semantically and
syntactically different samples. With only 2.5% of additional parameters,
MultiLoRA outperforms single LoRA counterparts and fine-tuning on multiple
benchmarks and model scales. Further investigation into weight update matrices
of MultiLoRA exhibits reduced dependency on top singular vectors and more
democratic unitary transform contributions.
655c194eb9d6d283fbedd5a5
Yiming Wang
admin_assigned
2023-11-21T11:24:28.333Z
62dce680aae4aa3ed0980de4
https://cdn-avatars.huggingface.co/v1/production/uploads/1658644064540-noauth.jpeg
yiming Wang
user
Alsace08
655c194eb9d6d283fbedd5a6
Yu Lin
655c194eb9d6d283fbedd5a7
Xiaodong Zeng
655c194eb9d6d283fbedd5a8
Guannan Zhang
MultiLoRA: Democratizing LoRA for Better Multi-Task Learning
LoRA achieves remarkable resource efficiency and comparable performance when
adapting LLMs for specific tasks. Since ChatGPT demonstrated superior
performance on various tasks, there has been a growing desire to adapt one
model for all tasks. However, the explicit low-rank of LoRA limits the
adaptation performance in complex multi-task scenarios. LoRA is dominated by a
small number of top singular vectors while fine-tuning decomposes into a set of
less important unitary transforms. In this paper, we propose MultiLoRA for
better multi-task adaptation by reducing the dominance of top singular vectors
observed in LoRA. MultiLoRA scales LoRA modules horizontally and change
parameter initialization of adaptation matrices to reduce parameter dependency,
thus yields more balanced unitary subspaces. We unprecedentedly construct
specialized training data by mixing datasets of instruction follow, natural
language understanding, world knowledge, to cover semantically and
syntactically different samples. With only 2.5% of additional parameters,
MultiLoRA outperforms single LoRA counterparts and fine-tuning on multiple
benchmarks and model scales. Further investigation into weight update matrices
of MultiLoRA exhibits reduced dependency on top singular vectors and more
democratic unitary transform contributions.
MultiLoRA enhances multi-task adaptation of LLMs by modifying LoRA modules, reducing dependency on top singular vectors, and achieving improved performance with minimal additional parameters.
LoRA LLMs ChatGPT low-rank fine-tuning unitary transforms MultiLoRA parameter initialization adaptation matrices weight update matrices singular vectors