LoGAH: Predicting 774-Million-Parameter Transformers using Graph
  HyperNetworks with 1/100 Parameters
A good initialization of deep learning models is essential since it can help
them converge better and faster. However, pretraining large models is
unaffordable for many researchers, which makes a desired prediction for initial
parameters more necessary nowadays. Graph HyperNetworks (GHNs), one approach to
predicting model parameters, have recently shown strong performance in
initializing large vision models. Unfortunately, predicting parameters of very
wide networks relies on copying small chunks of parameters multiple times and
requires an extremely large number of parameters to support full prediction,
which greatly hinders its adoption in practice. To address this limitation, we
propose LoGAH (Low-rank GrAph Hypernetworks), a GHN with a low-rank parameter
decoder that expands to significantly wider networks without requiring as
excessive increase of parameters as in previous attempts. LoGAH allows us to
predict the parameters of 774-million large neural networks in a
memory-efficient manner. We show that vision and language models (i.e., ViT and
GPT-2) initialized with LoGAH achieve better performance than those initialized
randomly or using existing hypernetworks. Furthermore, we show promising
transfer learning results w.r.t. training LoGAH on small datasets and using the
predicted parameters to initialize for larger tasks. We provide the codes in
https://github.com/Blackzxy/LoGAH .
66556035d200a2ecdfef4fd0
Xinyu Zhou
66556035d200a2ecdfef4fd1
Boris Knyazev
admin_assigned
2024-05-28T10:28:47.528Z
63e16165f0039731dfdd442a
/avatars/37cf99dc016c291c800f60d260173482.svg
Boris Knyazev
user
bknyaz
66556035d200a2ecdfef4fd2
Alexia Jolicoeur-Martineau
admin_assigned
2024-05-28T10:28:53.448Z
62f8ea1177b722f186611e8e
https://cdn-avatars.huggingface.co/v1/production/uploads/1660479589894-noauth.jpeg
Alexia Jolicoeur-Martineau
user
AlexiaJM
66556035d200a2ecdfef4fd3
Jie Fu
claimed_verified
2024-05-28T09:34:11.484Z
641a6895fb5ffff5ac79d593
https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/vxvwsto3llOEWGqQKGMYx.jpeg
Jie Fu
user
bigaidream
LoGAH: Predicting 774-Million-Parameter Transformers using Graph
  HyperNetworks with 1/100 Parameters
A good initialization of deep learning models is essential since it can help
them converge better and faster. However, pretraining large models is
unaffordable for many researchers, which makes a desired prediction for initial
parameters more necessary nowadays. Graph HyperNetworks (GHNs), one approach to
predicting model parameters, have recently shown strong performance in
initializing large vision models. Unfortunately, predicting parameters of very
wide networks relies on copying small chunks of parameters multiple times and
requires an extremely large number of parameters to support full prediction,
which greatly hinders its adoption in practice. To address this limitation, we
propose LoGAH (Low-rank GrAph Hypernetworks), a GHN with a low-rank parameter
decoder that expands to significantly wider networks without requiring as
excessive increase of parameters as in previous attempts. LoGAH allows us to
predict the parameters of 774-million large neural networks in a
memory-efficient manner. We show that vision and language models (i.e., ViT and
GPT-2) initialized with LoGAH achieve better performance than those initialized
randomly or using existing hypernetworks. Furthermore, we show promising
transfer learning results w.r.t. training LoGAH on small datasets and using the
predicted parameters to initialize for larger tasks. We provide the codes in
https://github.com/Blackzxy/LoGAH .
LoGAH, a low-rank parameter decoder Graph HyperNetwork, efficiently initializes large neural networks with better performance compared to random or existing hypernetworks.
Graph HyperNetworks GHNs LoGAH low-rank parameter decoder memory-efficient large neural networks vision models language models ViT GPT-2 transfer learning