Drop-Upcycling: Training Sparse Mixture of Experts with Partial
  Re-initialization
The Mixture of Experts (MoE) architecture reduces the training and inference
cost significantly compared to a dense model of equivalent capacity. Upcycling
is an approach that initializes and trains an MoE model using a pre-trained
dense model. While upcycling leads to initial performance gains, the training
progresses slower than when trained from scratch, leading to suboptimal
performance in the long term. We propose Drop-Upcycling - a method that
effectively addresses this problem. Drop-Upcycling combines two seemingly
contradictory approaches: utilizing the knowledge of pre-trained dense models
while statistically re-initializing some parts of the weights. This approach
strategically promotes expert specialization, significantly enhancing the MoE
model's efficiency in knowledge acquisition. Extensive large-scale experiments
demonstrate that Drop-Upcycling significantly outperforms previous MoE
construction methods in the long term, specifically when training on hundreds
of billions of tokens or more. As a result, our MoE model with 5.9B active
parameters achieves comparable performance to a 13B dense model in the same
model family, while requiring approximately 1/4 of the training FLOPs. All
experimental resources, including source code, training data, model checkpoints
and logs, are publicly available to promote reproducibility and future research
on MoE.
67c07170af68756abc571ab8
Taishi Nakamura
claimed_verified
2025-02-27T22:09:06.783Z
6308c49c454dc257521bc7f9
https://cdn-avatars.huggingface.co/v1/production/uploads/6308c49c454dc257521bc7f9/UWUS6OPa6OpVu1T0gd-wJ.jpeg
Taishi
user
Taishi-N324
67c07170af68756abc571ab9
Takuya Akiba
claimed_verified
2025-02-28T12:15:52.711Z
6482810dba6c556892f6f257
https://cdn-avatars.huggingface.co/v1/production/uploads/6482810dba6c556892f6f257/c7-wiVKenXiRtwnRpnjZN.jpeg
Takuya Akiba
user
iwiwi
67c07170af68756abc571aba
Kazuki Fujii
67c07170af68756abc571abb
Yusuke Oda
67c07170af68756abc571abc
Rio Yokota
67c07170af68756abc571abd
Jun Suzuki
Drop-Upcycling: Training Sparse Mixture of Experts with Partial
  Re-initialization
The Mixture of Experts (MoE) architecture reduces the training and inference
cost significantly compared to a dense model of equivalent capacity. Upcycling
is an approach that initializes and trains an MoE model using a pre-trained
dense model. While upcycling leads to initial performance gains, the training
progresses slower than when trained from scratch, leading to suboptimal
performance in the long term. We propose Drop-Upcycling - a method that
effectively addresses this problem. Drop-Upcycling combines two seemingly
contradictory approaches: utilizing the knowledge of pre-trained dense models
while statistically re-initializing some parts of the weights. This approach
strategically promotes expert specialization, significantly enhancing the MoE
model's efficiency in knowledge acquisition. Extensive large-scale experiments
demonstrate that Drop-Upcycling significantly outperforms previous MoE
construction methods in the long term, specifically when training on hundreds
of billions of tokens or more. As a result, our MoE model with 5.9B active
parameters achieves comparable performance to a 13B dense model in the same
model family, while requiring approximately 1/4 of the training FLOPs. All
experimental resources, including source code, training data, model checkpoints
and logs, are publicly available to promote reproducibility and future research
on MoE.
Mixture of Experts (MoE) upcycling pre-trained dense models Drop-Upcycling expert specialization knowledge acquisition FLOPs