Scaling Smart: Accelerating Large Language Model Pre-training with Small
  Model Initialization
The pre-training phase of language models often begins with randomly
initialized parameters. With the current trends in scaling models, training
their large number of parameters can be extremely slow and costly. In contrast,
small language models are less expensive to train, but they often cannot
achieve the accuracy of large models. In this paper, we explore an intriguing
idea to connect these two different regimes: Can we develop a method to
initialize large language models using smaller pre-trained models? Will such
initialization bring any benefits in terms of training time and final accuracy?
In this paper, we introduce HyperCloning, a method that can expand the
parameters of a pre-trained language model to those of a larger model with
increased hidden dimensions. Our method ensures that the larger model retains
the functionality of the smaller model. As a result, the larger model already
inherits the predictive power and accuracy of the smaller model before the
training starts. We demonstrate that training such an initialized model results
in significant savings in terms of GPU hours required for pre-training large
language models.
66ece1eca67d89bbe4284f8d
Mohammad Samragh
claimed_verified
2024-09-22T12:47:58.302Z
66eda2fefd187e5f9e4cd418
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/HcRJQ44HV5PeLSn5FoZ5G.png
Mohammad Samragh
user
msamragh
66ece1eca67d89bbe4284f8e
Iman Mirzadeh
admin_assigned
2024-09-20T09:24:44.826Z
66280a4c7c988f33b795f90f
/avatars/7a618b39c4ed20f5490ea64fbb67b5ac.svg
Iman Mirzadeh
user
imirzadeh-apple
66ece1eca67d89bbe4284f8f
Keivan Alizadeh Vahid
66ece1eca67d89bbe4284f90
Fartash Faghri
admin_assigned
2024-09-20T09:24:56.327Z
65f8945a3ba58880f9794a11
/avatars/fde0ea1c20f89b3a83dbefb24e728d61.svg
Fartash Faghri
user
fartashf
66ece1eca67d89bbe4284f91
Minsik Cho
admin_assigned
2024-09-20T09:25:06.825Z
630d422454c3dbd48058ba09
/avatars/1985c5812523fddc0835cde18a490856.svg
minsik cho
user
thyeros
66ece1eca67d89bbe4284f92
Moin Nabi
admin_assigned
2024-09-20T09:25:12.769Z
5fe36e9297b72d62ab9b90cd
/avatars/edfea7e8c86dcf355241f5a48365b35a.svg
Moin Nabi
user
moinnabi
66ece1eca67d89bbe4284f93
Devang Naik
66ece1eca67d89bbe4284f94
Mehrdad Farajtabar
admin_assigned
2024-09-20T09:25:23.623Z
62d188a6d50433c35eb14020
/avatars/fc3d149a27bc8396010a2a02148e8ca0.svg
Mehrdad Farajtabar
user
mfarajtabar
Scaling Smart: Accelerating Large Language Model Pre-training with Small
  Model Initialization
The pre-training phase of language models often begins with randomly
initialized parameters. With the current trends in scaling models, training
their large number of parameters can be extremely slow and costly. In contrast,
small language models are less expensive to train, but they often cannot
achieve the accuracy of large models. In this paper, we explore an intriguing
idea to connect these two different regimes: Can we develop a method to
initialize large language models using smaller pre-trained models? Will such
initialization bring any benefits in terms of training time and final accuracy?
In this paper, we introduce HyperCloning, a method that can expand the
parameters of a pre-trained language model to those of a larger model with
increased hidden dimensions. Our method ensures that the larger model retains
the functionality of the smaller model. As a result, the larger model already
inherits the predictive power and accuracy of the smaller model before the
training starts. We demonstrate that training such an initialized model results
in significant savings in terms of GPU hours required for pre-training large
language models.
HyperCloning expands smaller pre-trained language models to larger ones, retaining their accuracy and reducing training time.
HyperCloning parameter initialization large language models small language models pre-trained models training time GPU hours predictive power