Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and
  Mixture-of-Experts Optimization Alignment
While Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning for
Large Language Models (LLMs), its performance often falls short of Full
Fine-Tuning (Full FT). Current methods optimize LoRA by initializing with
static singular value decomposition (SVD) subsets, leading to suboptimal
leveraging of pre-trained knowledge. Another path for improving LoRA is
incorporating a Mixture-of-Experts (MoE) architecture. However, weight
misalignment and complex gradient dynamics make it challenging to adopt SVD
prior to the LoRA MoE architecture. To mitigate these issues, we propose
Great LoRA Mixture-of-Expert
(GOAT), a framework that (1) adaptively integrates relevant priors using an
SVD-structured MoE, and (2) aligns optimization with full fine-tuned MoE by
deriving a theoretical scaling factor. We demonstrate that proper scaling,
without modifying the architecture or training algorithms, boosts LoRA MoE's
efficiency and performance. Experiments across 25 datasets, including natural
language understanding, commonsense reasoning, image classification, and
natural language generation, demonstrate GOAT's state-of-the-art performance,
closing the gap with Full FT.
67bd396ea06bae99f3866911
Chenghao Fan
claimed_verified
2025-02-25T09:39:38.942Z
641aa5e391e3376a057bbd4c
/avatars/5818797f27444fde078b503774ee081c.svg
Chenghao Fan
user
Facico
67bd396ea06bae99f3866912
Zhenyi Lu
admin_assigned
2025-02-25T16:22:55.129Z
666b0a01e6cfd60425d00fd9
/avatars/f6e5447e95785563e850ffcbe7dd6e3d.svg
LUZHENYI
user
LUzhenyi123111
67bd396ea06bae99f3866913
Sichen Liu
claimed_verified
2025-02-26T08:37:56.904Z
641dbda28dc52733fa4419cf
https://cdn-avatars.huggingface.co/v1/production/uploads/641dbda28dc52733fa4419cf/vdIsa6UlMIaqHinGrYDb-.png
Sichen Liu
user
Seas0
67bd396ea06bae99f3866914
Xiaoye Qu
67bd396ea06bae99f3866915
Wei Wei
67bd396ea06bae99f3866916
Chengfeng Gu
admin_assigned
2025-02-25T16:23:29.392Z
675fc6bd34f2e646c06fbb07
/avatars/60f77c237cd652f80b7f1ecfd358afe1.svg
gu chengfeng
user
gucf
67bd396ea06bae99f3866917
Yu Cheng
claimed_verified
2025-06-02T07:53:02.137Z
67017abfe4d49b157ac534d9
/avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
Yu Cheng
user
ych133
Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and
  Mixture-of-Experts Optimization Alignment
While Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning for
Large Language Models (LLMs), its performance often falls short of Full
Fine-Tuning (Full FT). Current methods optimize LoRA by initializing with
static singular value decomposition (SVD) subsets, leading to suboptimal
leveraging of pre-trained knowledge. Another path for improving LoRA is
incorporating a Mixture-of-Experts (MoE) architecture. However, weight
misalignment and complex gradient dynamics make it challenging to adopt SVD
prior to the LoRA MoE architecture. To mitigate these issues, we propose
Great LoRA Mixture-of-Expert
(GOAT), a framework that (1) adaptively integrates relevant priors using an
SVD-structured MoE, and (2) aligns optimization with full fine-tuned MoE by
deriving a theoretical scaling factor. We demonstrate that proper scaling,
without modifying the architecture or training algorithms, boosts LoRA MoE's
efficiency and performance. Experiments across 25 datasets, including natural
language understanding, commonsense reasoning, image classification, and
natural language generation, demonstrate GOAT's state-of-the-art performance,
closing the gap with Full FT.
GOAT, a framework that integrates relevant priors using an SVD-structured Mixture-of-Experts and derives a theoretical scaling factor, enhances LoRA MoE's efficiency and performance, approaching Full Fine-Tuning results.
Low-Rank Adaptation LoRA Parameter-efficient fine-tuning Large Language Models Full Fine-Tuning Singular value decomposition SVD Mixture-of-Experts MoE Weight misalignment Gradient dynamics SVD-structured MoE Theoretical scaling factor