RandLoRA: Full-rank parameter-efficient fine-tuning of large models
Low-Rank Adaptation (LoRA) and its variants have shown impressive results in
reducing the number of trainable parameters and memory requirements of large
transformer networks while maintaining fine-tuning performance. However, the
low-rank nature of the weight update inherently limits the representation power
of fine-tuned models, potentially compromising performance on complex tasks.
This raises a critical question: when a performance gap between LoRA and
standard fine-tuning is observed, is it due to the reduced number of trainable
parameters or the rank deficiency? This paper aims to answer this question by
introducing RandLoRA, a parameter-efficient method that performs full-rank
updates using a learned linear combinations of low-rank, non-trainable random
matrices. Our method limits the number of trainable parameters by restricting
optimization to diagonal scaling matrices applied to the fixed random matrices.
This allows us to effectively overcome the low-rank limitations while
maintaining parameter and memory efficiency during training. Through extensive
experimentation across vision, language, and vision-language benchmarks, we
systematically evaluate the limitations of LoRA and existing random basis
methods. Our findings reveal that full-rank updates are beneficial across
vision and language tasks individually, and even more so for vision-language
tasks, where RandLoRA significantly reduces -- and sometimes eliminates -- the
performance gap between standard fine-tuning and LoRA, demonstrating its
efficacy.
67a1a4278b6584b24ff98eaf
Paul Albert
67a1a4278b6584b24ff98eb0
Frederic Z. Zhang
claimed_verified
2025-02-05T10:15:29.195Z
668ca1b3ed63008dfa687990
/avatars/b3ae0e7ff60c60db605288c9cf5ae6f3.svg
Fred Zhang
user
fredzzhang
67a1a4278b6584b24ff98eb1
Hemanth Saratchandran
67a1a4278b6584b24ff98eb2
Cristian Rodriguez-Opazo
67a1a4278b6584b24ff98eb3
Anton van den Hengel
67a1a4278b6584b24ff98eb4
Ehsan Abbasnejad
RandLoRA: Full-rank parameter-efficient fine-tuning of large models
Low-Rank Adaptation (LoRA) and its variants have shown impressive results in
reducing the number of trainable parameters and memory requirements of large
transformer networks while maintaining fine-tuning performance. However, the
low-rank nature of the weight update inherently limits the representation power
of fine-tuned models, potentially compromising performance on complex tasks.
This raises a critical question: when a performance gap between LoRA and
standard fine-tuning is observed, is it due to the reduced number of trainable
parameters or the rank deficiency? This paper aims to answer this question by
introducing RandLoRA, a parameter-efficient method that performs full-rank
updates using a learned linear combinations of low-rank, non-trainable random
matrices. Our method limits the number of trainable parameters by restricting
optimization to diagonal scaling matrices applied to the fixed random matrices.
This allows us to effectively overcome the low-rank limitations while
maintaining parameter and memory efficiency during training. Through extensive
experimentation across vision, language, and vision-language benchmarks, we
systematically evaluate the limitations of LoRA and existing random basis
methods. Our findings reveal that full-rank updates are beneficial across
vision and language tasks individually, and even more so for vision-language
tasks, where RandLoRA significantly reduces -- and sometimes eliminates -- the
performance gap between standard fine-tuning and LoRA, demonstrating its
efficacy.
RandLoRA addresses the limitations of LoRA by using full-rank updates with random matrices, improving performance on complex tasks without compromising parameter efficiency.
Low-Rank Adaptation LoRA parameter-efficient fine-tuning weight update representation power RandLoRA diagonal scaling matrices random matrices full-rank updates vision language vision-language benchmarks