LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for
  Parameter-Efficient Fine-Tuning
The rapid growth of model scale has necessitated substantial computational
resources for fine-tuning. Existing approach such as Low-Rank Adaptation (LoRA)
has sought to address the problem of handling the large updated parameters in
full fine-tuning. However, LoRA utilize random initialization and optimization
of low-rank matrices to approximate updated weights, which can result in
suboptimal convergence and an accuracy gap compared to full fine-tuning. To
address these issues, we propose LoLDU, a Parameter-Efficient Fine-Tuning
(PEFT) approach that significantly reduces trainable parameters by 2600 times
compared to regular PEFT methods while maintaining comparable performance.
LoLDU leverages Lower-Diag-Upper Decomposition (LDU) to initialize low-rank
matrices for faster convergence and orthogonality. We focus on optimizing the
diagonal matrix for scaling transformations. To the best of our knowledge,
LoLDU has the fewest parameters among all PEFT approaches. We conducted
extensive experiments across 4 instruction-following datasets, 6 natural
language understanding (NLU) datasets, 8 image classification datasets, and
image generation datasets with multiple model types (LLaMA2, RoBERTa, ViT, and
Stable Diffusion), providing a comprehensive and detailed analysis. Our
open-source code can be accessed at
https://github.com/SKDDJ/LoLDU{https://github.com/SKDDJ/LoLDU}.
6711dc54b265bbd413b8ec7b
Yiming Shi
extracted_confirmed
2024-10-18T04:03:46.223Z
642ebcefd09a9c63c2124bd2
https://cdn-avatars.huggingface.co/v1/production/uploads/642ebcefd09a9c63c2124bd2/HFSkoaxIM6nTq1CrY6jyF.jpeg
Yiming Shi
user
Shiym
6711dc54b265bbd413b8ec7c
Jiwei Wei
6711dc54b265bbd413b8ec7d
Yujia Wu
6711dc54b265bbd413b8ec7e
Ran Ran
6711dc54b265bbd413b8ec7f
Chengwei Sun
6711dc54b265bbd413b8ec80
Shiyuan He
6711dc54b265bbd413b8ec81
Yang Yang
LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for
  Parameter-Efficient Fine-Tuning
The rapid growth of model scale has necessitated substantial computational
resources for fine-tuning. Existing approach such as Low-Rank Adaptation (LoRA)
has sought to address the problem of handling the large updated parameters in
full fine-tuning. However, LoRA utilize random initialization and optimization
of low-rank matrices to approximate updated weights, which can result in
suboptimal convergence and an accuracy gap compared to full fine-tuning. To
address these issues, we propose LoLDU, a Parameter-Efficient Fine-Tuning
(PEFT) approach that significantly reduces trainable parameters by 2600 times
compared to regular PEFT methods while maintaining comparable performance.
LoLDU leverages Lower-Diag-Upper Decomposition (LDU) to initialize low-rank
matrices for faster convergence and orthogonality. We focus on optimizing the
diagonal matrix for scaling transformations. To the best of our knowledge,
LoLDU has the fewest parameters among all PEFT approaches. We conducted
extensive experiments across 4 instruction-following datasets, 6 natural
language understanding (NLU) datasets, 8 image classification datasets, and
image generation datasets with multiple model types (LLaMA2, RoBERTa, ViT, and
Stable Diffusion), providing a comprehensive and detailed analysis. Our
open-source code can be accessed at
https://github.com/SKDDJ/LoLDU{https://github.com/SKDDJ/LoLDU}.
LoLDU, a Parameter-Efficient Fine-Tuning approach using Lower-Diag-Upper Decomposition, significantly reduces trainable parameters while maintaining performance across various datasets and models.
Low-Rank Adaptation (LoRA) Parameter-Efficient Fine-Tuning (PEFT) Lower-Diag-Upper Decomposition (LDU) instruction-following datasets natural language understanding (NLU) datasets image classification datasets image generation datasets LLaMA2 RoBERTa ViT Stable Diffusion