Weight subcloning: direct initialization of transformers using larger
  pretrained ones
Training large transformer models from scratch for a target task requires
lots of data and is computationally demanding. The usual practice of transfer
learning overcomes this challenge by initializing the model with weights of a
pretrained model of the same size and specification to increase the convergence
and training speed. However, what if no pretrained model of the required size
is available? In this paper, we introduce a simple yet effective technique to
transfer the knowledge of a pretrained model to smaller variants. Our approach
called weight subcloning expedites the training of scaled-down transformers by
initializing their weights from larger pretrained models.
  Weight subcloning involves an operation on the pretrained model to obtain the
equivalent initialized scaled-down model. It consists of two key steps: first,
we introduce neuron importance ranking to decrease the embedding dimension per
layer in the pretrained model. Then, we remove blocks from the transformer
model to match the number of layers in the scaled-down network. The result is a
network ready to undergo training, which gains significant improvements in
training speed compared to random initialization. For instance, we achieve 4x
faster training for vision transformers in image classification and language
models designed for next token prediction.
657fa6ce17f67d5b8701c1f8
Mohammad Samragh
claimed_verified
2024-09-22T12:49:51.585Z
66eda2fefd187e5f9e4cd418
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/HcRJQ44HV5PeLSn5FoZ5G.png
Mohammad Samragh
user
msamragh
657fa6ce17f67d5b8701c1f9
Mehrdad Farajtabar
admin_assigned
2023-12-18T11:01:12.473Z
62d188a6d50433c35eb14020
/avatars/fc3d149a27bc8396010a2a02148e8ca0.svg
Mehrdad Farajtabar
user
mfarajtabar
657fa6ce17f67d5b8701c1fa
Sachin Mehta
admin_assigned
2023-12-18T11:01:34.876Z
64f792652003abb61bf5c1b8
/avatars/9721810de3519ece50b059c154baa9ca.svg
Sachin Mehta
user
sacmehta
657fa6ce17f67d5b8701c1fb
Raviteja Vemulapalli
657fa6ce17f67d5b8701c1fc
Fartash Faghri
claimed_verified
2024-04-26T15:29:01.893Z
65f8945a3ba58880f9794a11
/avatars/fde0ea1c20f89b3a83dbefb24e728d61.svg
Fartash Faghri
user
fartashf
657fa6ce17f67d5b8701c1fd
Devang Naik
657fa6ce17f67d5b8701c1fe
Oncel Tuzel
657fa6ce17f67d5b8701c1ff
Mohammad Rastegari
Weight subcloning: direct initialization of transformers using larger
  pretrained ones
Training large transformer models from scratch for a target task requires
lots of data and is computationally demanding. The usual practice of transfer
learning overcomes this challenge by initializing the model with weights of a
pretrained model of the same size and specification to increase the convergence
and training speed. However, what if no pretrained model of the required size
is available? In this paper, we introduce a simple yet effective technique to
transfer the knowledge of a pretrained model to smaller variants. Our approach
called weight subcloning expedites the training of scaled-down transformers by
initializing their weights from larger pretrained models.
  Weight subcloning involves an operation on the pretrained model to obtain the
equivalent initialized scaled-down model. It consists of two key steps: first,
we introduce neuron importance ranking to decrease the embedding dimension per
layer in the pretrained model. Then, we remove blocks from the transformer
model to match the number of layers in the scaled-down network. The result is a
network ready to undergo training, which gains significant improvements in
training speed compared to random initialization. For instance, we achieve 4x
faster training for vision transformers in image classification and language
models designed for next token prediction.
A technique called weight subcloning expedites the training of small transformer models using large pretrained models by reducing embedding dimensions and removing blocks.
transformer models transfer learning weight subcloning neuron importance ranking embedding dimension vision transformers language models next token prediction