Griffin: Mixing Gated Linear Recurrences with Local Attention for
  Efficient Language Models
Recurrent neural networks (RNNs) have fast inference and scale efficiently on
long sequences, but they are difficult to train and hard to scale. We propose
Hawk, an RNN with gated linear recurrences, and Griffin, a hybrid model that
mixes gated linear recurrences with local attention. Hawk exceeds the reported
performance of Mamba on downstream tasks, while Griffin matches the performance
of Llama-2 despite being trained on over 6 times fewer tokens. We also show
that Griffin can extrapolate on sequences significantly longer than those seen
during training. Our models match the hardware efficiency of Transformers
during training, and during inference they have lower latency and significantly
higher throughput. We scale Griffin up to 14B parameters, and explain how to
shard our models for efficient distributed training.
65e155a99944a44329e45362
Soham De
claimed_verified
2025-04-28T13:50:27.725Z
661415d9f9e58fe02bd8661d
/avatars/7a28bb507b8bac72a8b161e774b19fbd.svg
Soham De
user
sohamde
65e155a99944a44329e45363
Samuel L. Smith
65e155a99944a44329e45364
Anushan Fernando
65e155a99944a44329e45365
Aleksandar Botev
admin_assigned
2024-03-01T09:54:47.412Z
65d6647110d158dece540b5c
/avatars/fc0bbdeb0c23ceb9afbf2a658fc888f2.svg
Aleksandar Botev
user
botevmg
65e155a99944a44329e45366
George Cristian-Muraru
65e155a99944a44329e45367
Albert Gu
admin_assigned
2024-03-01T09:55:57.972Z
642619b62864ea9e59de834e
/avatars/a293aac4aa53800ad7dd08b28bd6b6f5.svg
Albert Gu
user
albertgu
65e155a99944a44329e45368
Ruba Haroun
65e155a99944a44329e45369
Leonard Berrada
65e155a99944a44329e4536a
Yutian Chen
65e155a99944a44329e4536b
Srivatsan Srinivasan
admin_assigned
2024-03-01T09:56:33.595Z
64f5121aa30d6b0a26c1b7e1
/avatars/862fe8ed200e4759835661bab161ad09.svg
Srivatsan Srinivasan
user
srivatr
65e155a99944a44329e4536c
Guillaume Desjardins
65e155a99944a44329e4536d
Arnaud Doucet
65e155a99944a44329e4536e
David Budden
65e155a99944a44329e4536f
Yee Whye Teh
65e155a99944a44329e45370
Razvan Pascanu
admin_assigned
2024-03-01T09:56:57.997Z
64b9310403124195cd9778ec
/avatars/57c594d3d0f97d3010b15b6a0806451c.svg
Razvan Pascanu
user
razp
65e155a99944a44329e45371
Nando De Freitas
65e155a99944a44329e45372
Caglar Gulcehre
admin_assigned
2024-03-01T09:57:08.266Z
64d28085db0b63b977a6c3f8
/avatars/994cc3ff7c18650fe2535e473d56b240.svg
Caglar Gulcehre
user
caglarg
Griffin: Mixing Gated Linear Recurrences with Local Attention for
  Efficient Language Models
Recurrent neural networks (RNNs) have fast inference and scale efficiently on
long sequences, but they are difficult to train and hard to scale. We propose
Hawk, an RNN with gated linear recurrences, and Griffin, a hybrid model that
mixes gated linear recurrences with local attention. Hawk exceeds the reported
performance of Mamba on downstream tasks, while Griffin matches the performance
of Llama-2 despite being trained on over 6 times fewer tokens. We also show
that Griffin can extrapolate on sequences significantly longer than those seen
during training. Our models match the hardware efficiency of Transformers
during training, and during inference they have lower latency and significantly
higher throughput. We scale Griffin up to 14B parameters, and explain how to
shard our models for efficient distributed training.
Hawk and Griffin, models incorporating gated linear recurrences, achieve high performance with lower computational resources and better hardware efficiency compared to existing architectures.
RNNs gated linear recurrences local attention Mamba Llama-2 tokens extrapolation hardware efficiency Transformers sharding