Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM
We investigate efficient methods for training Large Language Models (LLMs) to
possess capabilities in multiple specialized domains, such as coding, math
reasoning and world knowledge. Our method, named Branch-Train-MiX (BTX), starts
from a seed model, which is branched to train experts in embarrassingly
parallel fashion with high throughput and reduced communication cost. After
individual experts are asynchronously trained, BTX brings together their
feedforward parameters as experts in Mixture-of-Expert (MoE) layers and
averages the remaining parameters, followed by an MoE-finetuning stage to learn
token-level routing. BTX generalizes two special cases, the Branch-Train-Merge
method, which does not have the MoE finetuning stage to learn routing, and
sparse upcycling, which omits the stage of training experts asynchronously.
Compared to alternative approaches, BTX achieves the best accuracy-efficiency
tradeoff.
65f114b632c29f9fb7031dfa
Sainbayar Sukhbaatar
claimed_verified
2024-07-30T16:01:22.831Z
66a8611eb51510d82ed54231
/avatars/ad559e774fee4914091b82c9831ae2a2.svg
Sainbayar Sukhbaatar
user
sainbar
65f114b632c29f9fb7031dfb
Olga Golovneva
admin_assigned
2024-03-13T13:32:59.410Z
6318b1e06e6bd039440871d6
/avatars/2d9f96eb4092a9d1734dcc09303b259a.svg
Olga Golovneva
user
Golovneva
65f114b632c29f9fb7031dfc
Vasu Sharma
admin_assigned
2024-03-13T13:33:23.695Z
65492730a3d1682f79f3ab7a
/avatars/fc62605bdf10947dab393305c51ace96.svg
Vasu Sharma
user
vasusharma55
65f114b632c29f9fb7031dfd
Hu Xu
65f114b632c29f9fb7031dfe
Xi Victoria Lin
admin_assigned
2024-03-13T13:34:11.498Z
61df349e34b589fbaf2e93c4
/avatars/9dd7776d3b610d78e40c2e178ab2f112.svg
Victoria Lin
user
VictoriaLinML
65f114b632c29f9fb7031dff
Baptiste Rozière
admin_assigned
2024-03-13T13:34:21.092Z
62d063dac375d0c84255b9a1
/avatars/de0fc34bad8c761210c0895ebfa4feba.svg
Baptiste Roziere
user
broz
65f114b632c29f9fb7031e00
Jacob Kahn
claimed_verified
2024-03-13T19:44:10.442Z
65f1eac292b7b4b9d59cb886
/avatars/e7adb8a06860663c828501460cc092ae.svg
Jacob Kahn
user
jacobkahn
65f114b632c29f9fb7031e01
Daniel Li
65f114b632c29f9fb7031e02
Wen-tau Yih
65f114b632c29f9fb7031e03
Jason Weston
admin_assigned
2024-03-13T13:34:53.258Z
62f023a36a027498eaa2f9cc
/avatars/8ac1c5c74d0957e3c6cc94b3a7795c37.svg
Jason Weston
user
spermwhale
65f114b632c29f9fb7031e04
Xian Li
Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM
We investigate efficient methods for training Large Language Models (LLMs) to
possess capabilities in multiple specialized domains, such as coding, math
reasoning and world knowledge. Our method, named Branch-Train-MiX (BTX), starts
from a seed model, which is branched to train experts in embarrassingly
parallel fashion with high throughput and reduced communication cost. After
individual experts are asynchronously trained, BTX brings together their
feedforward parameters as experts in Mixture-of-Expert (MoE) layers and
averages the remaining parameters, followed by an MoE-finetuning stage to learn
token-level routing. BTX generalizes two special cases, the Branch-Train-Merge
method, which does not have the MoE finetuning stage to learn routing, and
sparse upcycling, which omits the stage of training experts asynchronously.
Compared to alternative approaches, BTX achieves the best accuracy-efficiency
tradeoff.
Branch-Train-MiX (BTX) method enhances Large Language Models by asynchronously training experts in parallel and integrating them using Mixture-of-Expert layers with token-level routing for improved accuracy and efficiency.
Large Language Models Branch-Train-MiX BTX seed model embarrassingly parallel Mixture-of-Expert MoE layers token-level routing Branch-Train-Merge sparse upcycling accuracy-efficiency tradeoff