Decoupling Angles and Strength in Low-rank Adaptation
Parameter-Efficient FineTuning (PEFT) methods have recently gained
significant popularity thanks to the widespread availability of large-scale
pretrained models. These methods allow for quick adaptation to downstream tasks
with minimal computational cost. However, popular finetuning methods such as
LoRA exhibit limited robustness when it comes to hyperparameter choices or
extended training regimes, preventing optimal out-of-the-box performance. In
contrast, bounded approaches, such as ETHER, provide greater robustness but are
limited to extremely low-rank adaptations and fixed-strength transformations,
reducing their adaptation expressive power. In this work, we propose Decoupled
Low-rank Adaptation (DeLoRA), a novel finetuning method that normalizes and
scales learnable low-rank matrices. By bounding the distance of the
transformation, DeLoRA effectively decouples the angular learning from the
adaptation strength, enhancing robustness without compromising performance.
Through evaluations on subject-driven image generation, natural language
understanding, and instruction tuning, we show that DeLoRA matches or surpasses
performance of competing PEFT methods, while exhibiting stronger robustness.
Code is available at https://github.com/ExplainableML/DeLoRA.
67ebabe3c545cab686735182
Massimo Bini
claimed_verified
2025-04-01T15:59:29.143Z
63f62ee3b29015adc33aafa0
/avatars/b7d7fe1b65333fb698402ed065fc5d13.svg
Massimo Bini
user
mwbini
67ebabe3c545cab686735183
Leander Girrbach
67ebabe3c545cab686735184
Zeynep Akata
Decoupling Angles and Strength in Low-rank Adaptation
Parameter-Efficient FineTuning (PEFT) methods have recently gained
significant popularity thanks to the widespread availability of large-scale
pretrained models. These methods allow for quick adaptation to downstream tasks
with minimal computational cost. However, popular finetuning methods such as
LoRA exhibit limited robustness when it comes to hyperparameter choices or
extended training regimes, preventing optimal out-of-the-box performance. In
contrast, bounded approaches, such as ETHER, provide greater robustness but are
limited to extremely low-rank adaptations and fixed-strength transformations,
reducing their adaptation expressive power. In this work, we propose Decoupled
Low-rank Adaptation (DeLoRA), a novel finetuning method that normalizes and
scales learnable low-rank matrices. By bounding the distance of the
transformation, DeLoRA effectively decouples the angular learning from the
adaptation strength, enhancing robustness without compromising performance.
Through evaluations on subject-driven image generation, natural language
understanding, and instruction tuning, we show that DeLoRA matches or surpasses
performance of competing PEFT methods, while exhibiting stronger robustness.
Code is available at https://github.com/ExplainableML/DeLoRA.
DeLoRA, a novel parameter-efficient fine-tuning method, enhances robustness and performance in various tasks compared to existing approaches by decoupling angular learning from adaptation strength.
Parameter-Efficient FineTuning (PEFT) LoRA bounded approaches ETHER Decoupled Low-rank Adaptation (DeLoRA) low-rank matrices robustness subject-driven image generation natural language understanding instruction tuning