Stack More Layers Differently: High-Rank Training Through Low-Rank
  Updates
Despite the dominance and effectiveness of scaling, resulting in large
networks with hundreds of billions of parameters, the necessity to train
overparametrized models remains poorly understood, and alternative approaches
do not necessarily make it cheaper to train high-performance models. In this
paper, we explore low-rank training techniques as an alternative approach to
training large neural networks. We introduce a novel method called ReLoRA,
which utilizes low-rank updates to train high-rank networks. We apply ReLoRA to
pre-training transformer language models with up to 350M parameters and
demonstrate comparable performance to regular neural network training.
Furthermore, we observe that the efficiency of ReLoRA increases with model
size, making it a promising approach for training multi-billion-parameter
networks efficiently. Our findings shed light on the potential of low-rank
training techniques and their implications for scaling laws.
64af5bc0398f0e7d07a97eb0
Vladislav Lialin
claimed_verified
2023-07-17T06:48:51.706Z
5f10910c636b661bdc42317e
https://cdn-avatars.huggingface.co/v1/production/uploads/1650850398598-5f10910c636b661bdc42317e.jpeg
Vlad Lialin
user
dropout05
64af5bc0398f0e7d07a97eb1
Namrata Shivagunde
64af5bc0398f0e7d07a97eb2
Sherin Muckatira
64af5bc0398f0e7d07a97eb3
Anna Rumshisky
Stack More Layers Differently: High-Rank Training Through Low-Rank
  Updates
Despite the dominance and effectiveness of scaling, resulting in large
networks with hundreds of billions of parameters, the necessity to train
overparametrized models remains poorly understood, and alternative approaches
do not necessarily make it cheaper to train high-performance models. In this
paper, we explore low-rank training techniques as an alternative approach to
training large neural networks. We introduce a novel method called ReLoRA,
which utilizes low-rank updates to train high-rank networks. We apply ReLoRA to
pre-training transformer language models with up to 350M parameters and
demonstrate comparable performance to regular neural network training.
Furthermore, we observe that the efficiency of ReLoRA increases with model
size, making it a promising approach for training multi-billion-parameter
networks efficiently. Our findings shed light on the potential of low-rank
training techniques and their implications for scaling laws.
ReLoRA, a low-rank training method, demonstrates comparable performance to traditional training for large transformer language models and becomes more efficient with increased model size.
low-rank training ReLoRA high-rank networks transformer language models scaling laws