LoRA Learns Less and Forgets Less
Low-Rank Adaptation (LoRA) is a widely-used parameter-efficient finetuning
method for large language models. LoRA saves memory by training only low rank
perturbations to selected weight matrices. In this work, we compare the
performance of LoRA and full finetuning on two target domains, programming and
mathematics. We consider both the instruction finetuning (approx100K
prompt-response pairs) and continued pretraining (approx10B unstructured
tokens) data regimes. Our results show that, in most settings, LoRA
substantially underperforms full finetuning. Nevertheless, LoRA exhibits a
desirable form of regularization: it better maintains the base model's
performance on tasks outside the target domain. We show that LoRA provides
stronger regularization compared to common techniques such as weight decay and
dropout; it also helps maintain more diverse generations. We show that full
finetuning learns perturbations with a rank that is 10-100X greater than
typical LoRA configurations, possibly explaining some of the reported gaps. We
conclude by proposing best practices for finetuning with LoRA.
6646bb2f9d3a5b2ddb400cf3
Dan Biderman
admin_assigned
2024-05-17T09:19:52.289Z
6478a53d79f2d49511eda935
/avatars/9ca81c92bee68022b68c2ab9315135ef.svg
Dan Biderman
user
danbider
6646bb2f9d3a5b2ddb400cf4
Jose Gonzalez Ortiz
6646bb2f9d3a5b2ddb400cf5
Jacob Portes
admin_assigned
2024-05-17T09:21:10.974Z
63dd38e5a8877129a15a12c5
https://cdn-avatars.huggingface.co/v1/production/uploads/1676407968549-63dd38e5a8877129a15a12c5.png
Jacob Portes
user
jacobfulano
6646bb2f9d3a5b2ddb400cf6
Mansheej Paul
admin_assigned
2024-05-17T09:21:16.531Z
622a8e4c9f6139daa8e99815
/avatars/d9beaabef72d89fe3c6a43937b78d977.svg
Mansheej Paul
user
mansheej
6646bb2f9d3a5b2ddb400cf7
Philip Greengard
6646bb2f9d3a5b2ddb400cf8
Connor Jennings
admin_assigned
2024-05-17T09:26:02.830Z
64892b0135f2eb53106a8cce
/avatars/c6c69e8fb4e0a2f46c1f98ba439ea92f.svg
Connor Jennings
user
cojenninmosaic
6646bb2f9d3a5b2ddb400cf9
Daniel King
6646bb2f9d3a5b2ddb400cfa
Sam Havens
admin_assigned
2024-05-17T09:25:42.054Z
6374382eb2484d1af7046cdf
https://cdn-avatars.huggingface.co/v1/production/uploads/1668560930781-noauth.png
Sam Havens
user
sam-mosaic
6646bb2f9d3a5b2ddb400cfb
Vitaliy Chiley
admin_assigned
2024-05-17T09:26:09.014Z
6451a50e67703e98a3cd7835
https://cdn-avatars.huggingface.co/v1/production/uploads/6451a50e67703e98a3cd7835/gsCQA_5p735o0akwqxc5B.png
Vitaliy
user
vchiley
6646bb2f9d3a5b2ddb400cfc
Jonathan Frankle
admin_assigned
2024-05-17T09:26:16.892Z
62b0b7f9a596c50b1783fe9e
https://cdn-avatars.huggingface.co/v1/production/uploads/62b0b7f9a596c50b1783fe9e/nzctJXoyiEwCnOHVJEx3_.png
Jonathan Frankle
user
jfrankle
6646bb2f9d3a5b2ddb400cfd
Cody Blakeney
admin_assigned
2024-05-17T09:26:32.204Z
642cc235b09c70b36de2be78
https://cdn-avatars.huggingface.co/v1/production/uploads/642cc235b09c70b36de2be78/K2P1S3Quw1btoJIhRs6j9.jpeg
Cody Blakeney
user
codestar12
6646bb2f9d3a5b2ddb400cfe
John P. Cunningham
LoRA Learns Less and Forgets Less
Low-Rank Adaptation (LoRA) is a widely-used parameter-efficient finetuning
method for large language models. LoRA saves memory by training only low rank
perturbations to selected weight matrices. In this work, we compare the
performance of LoRA and full finetuning on two target domains, programming and
mathematics. We consider both the instruction finetuning (approx100K
prompt-response pairs) and continued pretraining (approx10B unstructured
tokens) data regimes. Our results show that, in most settings, LoRA
substantially underperforms full finetuning. Nevertheless, LoRA exhibits a
desirable form of regularization: it better maintains the base model's
performance on tasks outside the target domain. We show that LoRA provides
stronger regularization compared to common techniques such as weight decay and
dropout; it also helps maintain more diverse generations. We show that full
finetuning learns perturbations with a rank that is 10-100X greater than
typical LoRA configurations, possibly explaining some of the reported gaps. We
conclude by proposing best practices for finetuning with LoRA.
LoRA, a parameter-efficient finetuning method for large language models, underperforms full finetuning in target domains but provides better regularization and maintains diverse generation compared to other techniques.
Low-Rank Adaptation LoRA parameter-efficient finetuning full finetuning instruction finetuning continued pretraining regularization base model performance weight decay dropout diverse generations