BAM! Just Like That: Simple and Efficient Parameter Upcycling for
  Mixture of Experts
The Mixture of Experts (MoE) framework has become a popular architecture for
large language models due to its superior performance over dense models.
However, training MoEs from scratch in a large-scale regime is prohibitively
expensive. Existing methods mitigate this by pre-training multiple dense expert
models independently and using them to initialize an MoE. This is done by using
experts' feed-forward network (FFN) to initialize the MoE's experts while
merging other parameters. However, this method limits the reuse of dense model
parameters to only the FFN layers, thereby constraining the advantages when
"upcycling" these models into MoEs. We propose BAM (Branch-Attend-Mix), a
simple yet effective method that addresses this shortcoming. BAM makes full use
of specialized dense models by not only using their FFN to initialize the MoE
layers but also leveraging experts' attention parameters fully by initializing
them into a soft-variant of Mixture of Attention (MoA) layers. We explore two
methods for upcycling attention parameters: 1) initializing separate attention
experts from dense models including all attention parameters for the best model
performance; and 2) sharing key and value parameters across all experts to
facilitate for better inference efficiency. To further improve efficiency, we
adopt a parallel attention transformer architecture to MoEs, which allows the
attention experts and FFN experts to be computed concurrently. Our experiments
on seed models ranging from 590 million to 2 billion parameters demonstrate
that BAM surpasses baselines in both perplexity and downstream task
performance, within the same computational and data constraints.
66bed259a9425c872d3eaedb
Qizhen Zhang
66bed259a9425c872d3eaedc
Nikolas Gritsch
66bed259a9425c872d3eaedd
Dwaraknath Gnaneshwar
66bed259a9425c872d3eaede
Simon Guo
claimed_verified
2025-02-25T09:42:25.438Z
65ef61af8179815f59d189ea
https://cdn-avatars.huggingface.co/v1/production/uploads/65ef61af8179815f59d189ea/fknSvriVyP4KMI7Z53LNH.png
Simon Guo
user
simonguozirui
66bed259a9425c872d3eaedf
David Cairuz
66bed259a9425c872d3eaee0
Bharat Venkitesh
66bed259a9425c872d3eaee1
Jakob Foerster
66bed259a9425c872d3eaee2
Phil Blunsom
66bed259a9425c872d3eaee3
Sebastian Ruder
66bed259a9425c872d3eaee4
Ahmet Ustun
66bed259a9425c872d3eaee5
Acyr Locatelli
BAM! Just Like That: Simple and Efficient Parameter Upcycling for
  Mixture of Experts
The Mixture of Experts (MoE) framework has become a popular architecture for
large language models due to its superior performance over dense models.
However, training MoEs from scratch in a large-scale regime is prohibitively
expensive. Existing methods mitigate this by pre-training multiple dense expert
models independently and using them to initialize an MoE. This is done by using
experts' feed-forward network (FFN) to initialize the MoE's experts while
merging other parameters. However, this method limits the reuse of dense model
parameters to only the FFN layers, thereby constraining the advantages when
"upcycling" these models into MoEs. We propose BAM (Branch-Attend-Mix), a
simple yet effective method that addresses this shortcoming. BAM makes full use
of specialized dense models by not only using their FFN to initialize the MoE
layers but also leveraging experts' attention parameters fully by initializing
them into a soft-variant of Mixture of Attention (MoA) layers. We explore two
methods for upcycling attention parameters: 1) initializing separate attention
experts from dense models including all attention parameters for the best model
performance; and 2) sharing key and value parameters across all experts to
facilitate for better inference efficiency. To further improve efficiency, we
adopt a parallel attention transformer architecture to MoEs, which allows the
attention experts and FFN experts to be computed concurrently. Our experiments
on seed models ranging from 590 million to 2 billion parameters demonstrate
that BAM surpasses baselines in both perplexity and downstream task
performance, within the same computational and data constraints.
BAM enhances the Mixture of Experts framework by fully recycling dense model parameters, including attention layers, leading to better performance and efficiency than previous methods.
Mixture of Experts dense models feed-forward network initialization attention parameters Mixture of Attention parallel attention transformer perplexity downstream tasks