VeRA: Vector-based Random Matrix Adaptation
Low-rank adapation (LoRA) is a popular method that reduces the number of
trainable parameters when finetuning large language models, but still faces
acute storage challenges when scaling to even larger models or deploying
numerous per-user or per-task adapted models. In this work, we present
Vector-based Random Matrix Adaptation (VeRA), which reduces the number of
trainable parameters by 10x compared to LoRA, yet maintains the same
performance. It achieves this by using a single pair of low-rank matrices
shared across all layers and learning small scaling vectors instead. We
demonstrate its effectiveness on the GLUE and E2E benchmarks, and show its
application in instruction-following with just 1.4M parameters using the Llama2
7B model.
652f5762660516114945795c
Dawid Jan Kopiczko
extracted_confirmed
2023-10-18T10:50:02.300Z
637de0fa1342ba1762422495
/avatars/b1cce386a8b33007fc381fcbfc5cdc9a.svg
Dawid
user
dakopi
652f5762660516114945795d
Tijmen Blankevoort
652f5762660516114945795e
Yuki Markus Asano
claimed_verified
2023-10-18T09:07:54.798Z
637d21239a5217b88b7549c3
https://cdn-avatars.huggingface.co/v1/production/uploads/637d21239a5217b88b7549c3/LrIGPiva5VGVZG87rTAJz.jpeg
Yuki Asano
user
yukimasano
VeRA: Vector-based Random Matrix Adaptation
Low-rank adapation (LoRA) is a popular method that reduces the number of
trainable parameters when finetuning large language models, but still faces
acute storage challenges when scaling to even larger models or deploying
numerous per-user or per-task adapted models. In this work, we present
Vector-based Random Matrix Adaptation (VeRA), which reduces the number of
trainable parameters by 10x compared to LoRA, yet maintains the same
performance. It achieves this by using a single pair of low-rank matrices
shared across all layers and learning small scaling vectors instead. We
demonstrate its effectiveness on the GLUE and E2E benchmarks, and show its
application in instruction-following with just 1.4M parameters using the Llama2
7B model.
Low-rank adapation (LoRA) Vector-based Random Matrix Adaptation (VeRA) low-rank matrices scaling vectors GLUE benchmark E2E benchmark instruction-following