Zamba: A Compact 7B SSM Hybrid Model
In this technical report, we present Zamba, a novel 7B SSM-transformer hybrid
model which achieves competitive performance against leading open-weight models
at a comparable scale. Zamba is trained on 1T tokens from openly available
datasets and is the best non-transformer model at this scale. Zamba pioneers a
unique architecture combining a Mamba backbone with a single shared attention
module, thus obtaining the benefits of attention at minimal parameter cost. Due
to its architecture, Zamba is significantly faster at inference than comparable
transformer models and requires substantially less memory for generation of
long sequences. Zamba is pretrained in two phases: the first phase is based on
existing web datasets, while the second one consists of annealing the model
over high-quality instruct and synthetic datasets, and is characterized by a
rapid learning rate decay. We open-source the weights and all checkpoints for
Zamba, through both phase 1 and annealing phases.
66555178c3b51bcf54afc562
Paolo Glorioso
admin_assigned
2024-05-28T09:40:40.762Z
64e40335c0edca443ef8af3e
/avatars/c42a72a448dcb9e8bb3a0b41d1b6397a.svg
Paolo Glorioso
user
pglo
66555178c3b51bcf54afc563
Quentin Anthony
admin_assigned
2024-05-28T09:47:35.906Z
65bc13717c6ad1994b6619e9
/avatars/8e97e25e2de30f6bea9f9bc94418ff0f.svg
Quentin Anthony
user
qanthony-z
66555178c3b51bcf54afc564
Yury Tokpanov
admin_assigned
2024-05-28T09:47:43.700Z
65455aca468722e935103b17
/avatars/29786a7df6541aa1e5d408eb06b0bd96.svg
Yury Tokpanov
user
yury-zyphra
66555178c3b51bcf54afc565
James Whittington
66555178c3b51bcf54afc566
Jonathan Pilault
admin_assigned
2024-05-28T09:48:00.516Z
65f1a7b2ea222a82562de3e9
/avatars/7f62489688acf30d4edc7f9d2036b14b.svg
Jonathan Pilault
user
jpilault
66555178c3b51bcf54afc567
Adam Ibrahim
admin_assigned
2024-05-28T09:48:06.540Z
648743f05aed4a7b0bf2e3a6
/avatars/e6f3b35085b1b8cb2c231516d1fea70a.svg
Adam Ibrahim
user
adami1
66555178c3b51bcf54afc568
Beren Millidge
admin_assigned
2024-05-28T09:48:12.249Z
65c05e75c084467acab2f84a
/avatars/255e98fe436d9d9252975e842755c0ff.svg
Beren Millidge
user
BerenMillidge
Zamba: A Compact 7B SSM Hybrid Model
In this technical report, we present Zamba, a novel 7B SSM-transformer hybrid
model which achieves competitive performance against leading open-weight models
at a comparable scale. Zamba is trained on 1T tokens from openly available
datasets and is the best non-transformer model at this scale. Zamba pioneers a
unique architecture combining a Mamba backbone with a single shared attention
module, thus obtaining the benefits of attention at minimal parameter cost. Due
to its architecture, Zamba is significantly faster at inference than comparable
transformer models and requires substantially less memory for generation of
long sequences. Zamba is pretrained in two phases: the first phase is based on
existing web datasets, while the second one consists of annealing the model
over high-quality instruct and synthetic datasets, and is characterized by a
rapid learning rate decay. We open-source the weights and all checkpoints for
Zamba, through both phase 1 and annealing phases.
Zamba, a 7B SSM-transformer hybrid model, achieves competitive performance with minimal parameter cost and efficient inference by combining a Mamba backbone with a shared attention module.
SSM-transformer Mamba backbone shared attention module inference speed memory efficiency pretraining annealing instruct datasets synthetic datasets rapid learning rate decay