VeRA: Vector-based Random Matrix Adaptation
Low-rank adapation (LoRA) is a popular method that reduces the number of
trainable parameters when finetuning large language models, but still faces
acute storage challenges when scaling to even larger models or deploying
numerous per-user or per-task adapted models. In this work, we present
Vector-based Random Matrix Adaptation (VeRA), which reduces the number of
trainable parameters by 10x compared to LoRA, yet maintains the same
performance. It achieves this by using a single pair of low-rank matrices
shared across all layers and learning small scaling vectors instead. We
demonstrate its effectiveness on the GLUE and E2E benchmarks, and show its
application in instruction-following with just 1.4M parameters using the Llama2
7B model.
652f5762660516114945795c
Dawid Jan Kopiczko
extracted_confirmed
2023-10-18T10:50:02.300Z
637de0fa1342ba1762422495
/avatars/b1cce386a8b33007fc381fcbfc5cdc9a.svg
Dawid
user
dakopi
652f5762660516114945795d
Tijmen Blankevoort
652f5762660516114945795e
Yuki Markus Asano
claimed_verified
2023-10-18T09:07:54.798Z
637d21239a5217b88b7549c3
https://cdn-avatars.huggingface.co/v1/production/uploads/637d21239a5217b88b7549c3/LrIGPiva5VGVZG87rTAJz.jpeg
Yuki Asano
user
yukimasano
VeRA: Vector-based Random Matrix Adaptation
Low-rank adapation (LoRA) is a popular method that reduces the number of
trainable parameters when finetuning large language models, but still faces
acute storage challenges when scaling to even larger models or deploying
numerous per-user or per-task adapted models. In this work, we present
Vector-based Random Matrix Adaptation (VeRA), which reduces the number of
trainable parameters by 10x compared to LoRA, yet maintains the same
performance. It achieves this by using a single pair of low-rank matrices
shared across all layers and learning small scaling vectors instead. We
demonstrate its effectiveness on the GLUE and E2E benchmarks, and show its
application in instruction-following with just 1.4M parameters using the Llama2
7B model.
Vector-based Random Matrix Adaptation (VeRA) reduces the number of trainable parameters by 10x compared to LoRA while maintaining performance, and is demonstrated on benchmarks like GLUE and E2E, showing its utility in instruction-following.
Low-rank adaptation LoRA Vector-based Random Matrix Adaptation VeRA GLUE E2E instruction-following Llama2