BlackMamba: Mixture of Experts for State-Space Models
State-space models (SSMs) have recently demonstrated competitive performance
to transformers at large-scale language modeling benchmarks while achieving
linear time and memory complexity as a function of sequence length. Mamba, a
recently released SSM model, shows impressive performance in both language
modeling and long sequence processing tasks. Simultaneously, mixture-of-expert
(MoE) models have shown remarkable performance while significantly reducing the
compute and latency costs of inference at the expense of a larger memory
footprint. In this paper, we present BlackMamba, a novel architecture that
combines the Mamba SSM with MoE to obtain the benefits of both. We demonstrate
that BlackMamba performs competitively against both Mamba and transformer
baselines, and outperforms in inference and training FLOPs. We fully train and
open-source 340M/1.5B and 630M/2.8B BlackMamba models on 300B tokens of a
custom dataset. We show that BlackMamba inherits and combines both of the
benefits of SSM and MoE architectures, combining linear-complexity generation
from SSM with cheap and fast inference from MoE. We release all weights,
checkpoints, and inference code open-source. Inference code at:
https://github.com/Zyphra/BlackMamba
65c1b21bca9755314bf8178c
Quentin Anthony
extracted_confirmed
2024-02-06T17:52:09.150Z
65bc13717c6ad1994b6619e9
/avatars/8e97e25e2de30f6bea9f9bc94418ff0f.svg
Quentin Anthony
user
qanthony-z
65c1b21bca9755314bf8178d
Yury Tokpanov
extracted_pending
2024-02-06T04:14:19.754Z
65455aca468722e935103b17
/avatars/29786a7df6541aa1e5d408eb06b0bd96.svg
Yury Tokpanov
user
yury-zyphra
65c1b21bca9755314bf8178e
Paolo Glorioso
65c1b21bca9755314bf8178f
Beren Millidge
claimed_verified
2024-08-13T07:41:27.149Z
65c05e75c084467acab2f84a
/avatars/255e98fe436d9d9252975e842755c0ff.svg
Beren Millidge
user
BerenMillidge
BlackMamba: Mixture of Experts for State-Space Models
State-space models (SSMs) have recently demonstrated competitive performance
to transformers at large-scale language modeling benchmarks while achieving
linear time and memory complexity as a function of sequence length. Mamba, a
recently released SSM model, shows impressive performance in both language
modeling and long sequence processing tasks. Simultaneously, mixture-of-expert
(MoE) models have shown remarkable performance while significantly reducing the
compute and latency costs of inference at the expense of a larger memory
footprint. In this paper, we present BlackMamba, a novel architecture that
combines the Mamba SSM with MoE to obtain the benefits of both. We demonstrate
that BlackMamba performs competitively against both Mamba and transformer
baselines, and outperforms in inference and training FLOPs. We fully train and
open-source 340M/1.5B and 630M/2.8B BlackMamba models on 300B tokens of a
custom dataset. We show that BlackMamba inherits and combines both of the
benefits of SSM and MoE architectures, combining linear-complexity generation
from SSM with cheap and fast inference from MoE. We release all weights,
checkpoints, and inference code open-source. Inference code at:
https://github.com/Zyphra/BlackMamba
BlackMamba, a novel architecture combining Mamba SSM and MoE models, achieves competitive performance in language modeling and improved inference and training efficiency.
state-space models SSM Mamba mixture-of-experts MoE BlackMamba linear time and memory complexity language modeling long sequence processing inference FLOPs training FLOPs custom dataset linear-complexity generation cheap and fast inference