One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning
We present Generalized LoRA (GLoRA), an advanced approach for universal
parameter-efficient fine-tuning tasks. Enhancing Low-Rank Adaptation (LoRA),
GLoRA employs a generalized prompt module to optimize pre-trained model weights
and adjust intermediate activations, providing more flexibility and capability
across diverse tasks and datasets. Moreover, GLoRA facilitates efficient
parameter adaptation by employing a scalable, modular, layer-wise structure
search that learns individual adapter of each layer. Originating from a unified
mathematical formulation, GLoRA exhibits strong transfer learning, few-shot
learning and domain generalization abilities, as it adjusts to new tasks
through additional dimensions on weights and activations. Comprehensive
experiments demonstrate that GLoRA outperforms all previous methods in natural,
specialized, and structured benchmarks, achieving superior accuracy with fewer
parameters and computations on various datasets. Furthermore, our structural
re-parameterization design ensures that GLoRA incurs no extra inference cost,
rendering it a practical solution for resource-limited applications. Code is
available at: https://github.com/Arnav0400/ViT-Slim/tree/master/GLoRA.
648938c3de81ed071f971cf2
Arnav Chavan
claimed_verified
2023-06-14T09:29:11.295Z
648979dd308f19bc7c42899f
/avatars/6b7bde1d453f0328fcd5a86cfd9eb3b2.svg
Arnav Chavan
user
Arnav0400
648938c3de81ed071f971cf3
Zhuang Liu
claimed_verified
2024-02-21T19:38:50.005Z
647d3a9383cdb93baf76dce2
/avatars/56efca912a66ef9d4f2f3679aac31bae.svg
Zhuang Liu
user
liuzhuang13
648938c3de81ed071f971cf4
Deepak Gupta
648938c3de81ed071f971cf5
Eric Xing
648938c3de81ed071f971cf6
Zhiqiang Shen
One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning
We present Generalized LoRA (GLoRA), an advanced approach for universal
parameter-efficient fine-tuning tasks. Enhancing Low-Rank Adaptation (LoRA),
GLoRA employs a generalized prompt module to optimize pre-trained model weights
and adjust intermediate activations, providing more flexibility and capability
across diverse tasks and datasets. Moreover, GLoRA facilitates efficient
parameter adaptation by employing a scalable, modular, layer-wise structure
search that learns individual adapter of each layer. Originating from a unified
mathematical formulation, GLoRA exhibits strong transfer learning, few-shot
learning and domain generalization abilities, as it adjusts to new tasks
through additional dimensions on weights and activations. Comprehensive
experiments demonstrate that GLoRA outperforms all previous methods in natural,
specialized, and structured benchmarks, achieving superior accuracy with fewer
parameters and computations on various datasets. Furthermore, our structural
re-parameterization design ensures that GLoRA incurs no extra inference cost,
rendering it a practical solution for resource-limited applications. Code is
available at: https://github.com/Arnav0400/ViT-Slim/tree/master/GLoRA.
GLoRA, an advanced method for parameter-efficient fine-tuning, enhances LoRA with a generalized prompt module and modular layer-wise structure search, offering superior performance across diverse tasks with fewer parameters and computational costs.
Generalized LoRA GLoRA Low-Rank Adaptation prompt module pre-trained model weights intermediate activations parameter adaptation unified mathematical formulation transfer learning few-shot learning domain generalization structural re-parameterization inference cost