Recycling Pretrained Checkpoints: Orthogonal Growth of
  Mixture-of-Experts for Efficient Large Language Model Pre-Training
The rapidly increasing computational cost of pretraining Large Language
Models necessitates more efficient approaches. Numerous computational costs
have been invested in existing well-trained checkpoints, but many of them
remain underutilized due to engineering constraints or limited model capacity.
To efficiently reuse this "sunk" cost, we propose to recycle pretrained
checkpoints by expanding their parameter counts and continuing training. We
propose orthogonal growth method well-suited for converged Mixture-of-Experts
model: interpositional layer copying for depth growth and expert duplication
with injected noise for width growth. To determine the optimal timing for such
growth across checkpoints sequences, we perform comprehensive scaling
experiments revealing that the final accuracy has a strong positive correlation
with the amount of sunk cost, indicating that greater prior investment leads to
better performance. We scale our approach to models with 70B parameters and
over 1T training tokens, achieving 10.66% accuracy gain over training from
scratch under the same additional compute budget. Our checkpoint recycling
approach establishes a foundation for economically efficient large language
model pretraining.
68e86f1b95e8e6771df38945
Ruizhe Wang
claimed_verified
2025-10-10T04:32:36.152Z
63203d4e260e691cfc19fcb1
/avatars/72437259c73cc4a950a2e84141097310.svg
Ruizhe Wang
user
Mr-Philo
68e86f1b95e8e6771df38946
Yucheng Ding
68e86f1b95e8e6771df38947
Xiao Liu
claimed_verified
2025-10-10T09:12:18.682Z
63fb6e281b4b1bd4e7ffc5be
https://cdn-avatars.huggingface.co/v1/production/uploads/63fb6e281b4b1bd4e7ffc5be/aiRu_bulgnxvEMrjipGoQ.jpeg
Xiao Liu
user
lx865712528
68e86f1b95e8e6771df38948
Yaoxiang Wang
68e86f1b95e8e6771df38949
Peng Cheng
68e86f1b95e8e6771df3894a
Baining Guo
68e86f1b95e8e6771df3894b
Zhengjun Zha
68e86f1b95e8e6771df3894c
Yeyun Gong
Recycling Pretrained Checkpoints: Orthogonal Growth of
  Mixture-of-Experts for Efficient Large Language Model Pre-Training
The rapidly increasing computational cost of pretraining Large Language
Models necessitates more efficient approaches. Numerous computational costs
have been invested in existing well-trained checkpoints, but many of them
remain underutilized due to engineering constraints or limited model capacity.
To efficiently reuse this "sunk" cost, we propose to recycle pretrained
checkpoints by expanding their parameter counts and continuing training. We
propose orthogonal growth method well-suited for converged Mixture-of-Experts
model: interpositional layer copying for depth growth and expert duplication
with injected noise for width growth. To determine the optimal timing for such
growth across checkpoints sequences, we perform comprehensive scaling
experiments revealing that the final accuracy has a strong positive correlation
with the amount of sunk cost, indicating that greater prior investment leads to
better performance. We scale our approach to models with 70B parameters and
over 1T training tokens, achieving 10.66% accuracy gain over training from
scratch under the same additional compute budget. Our checkpoint recycling
approach establishes a foundation for economically efficient large language
model pretraining.
Recycling pretrained checkpoints through orthogonal growth methods improves large language model performance with reduced computational cost.
Large Language Models pretrained checkpoints orthogonal growth method interpositional layer copying expert duplication injected noise scaling experiments Mixture-of-Experts model