RecurrentGemma: Moving Past Transformers for Efficient Open Language
  Models
We introduce RecurrentGemma, an open language model which uses Google's novel
Griffin architecture. Griffin combines linear recurrences with local attention
to achieve excellent performance on language. It has a fixed-sized state, which
reduces memory use and enables efficient inference on long sequences. We
provide a pre-trained model with 2B non-embedding parameters, and an
instruction tuned variant. Both models achieve comparable performance to
Gemma-2B despite being trained on fewer tokens.
661890a365e0419878fcb603
Aleksandar Botev
admin_assigned
2024-04-18T12:01:11.116Z
65d6647110d158dece540b5c
/avatars/fc0bbdeb0c23ceb9afbf2a658fc888f2.svg
Aleksandar Botev
user
botevmg
661890a365e0419878fcb604
Soham De
admin_assigned
2024-04-18T12:01:25.519Z
661415d9f9e58fe02bd8661d
/avatars/7a28bb507b8bac72a8b161e774b19fbd.svg
Soham De
user
sohamde
661890a365e0419878fcb605
Samuel L Smith
661890a365e0419878fcb606
Anushan Fernando
admin_assigned
2024-04-18T12:01:44.512Z
660c2ccb3aa98dc4b27de7b3
/avatars/2b19094e1b876a1be1f763dc941ff0bf.svg
Anushan Fernando
user
AnushanF
661890a365e0419878fcb607
George-Cristian Muraru
admin_assigned
2024-04-18T12:01:50.783Z
660dd120fc9742e05187cffd
https://cdn-avatars.huggingface.co/v1/production/uploads/660dd120fc9742e05187cffd/bFZHJMJcN6rJGo_Yala89.png
George-Cristian Muraru
user
gmuraru
661890a365e0419878fcb608
Ruba Haroun
admin_assigned
2024-04-18T12:01:57.539Z
66167eec24c94fd8a2427513
/avatars/24ff7acfbf064a3a37dcad25ef9d45ee.svg
Ruba Haroun
user
RubaHaroun
661890a365e0419878fcb609
Leonard Berrada
admin_assigned
2024-04-18T12:02:04.421Z
65fc77aad6206abe6cb611f1
/avatars/306334124d90d1b4d4f82fb47f56156f.svg
Leonard Berrada
user
lberrada
661890a365e0419878fcb60a
Razvan Pascanu
admin_assigned
2024-04-18T12:02:10.349Z
64b9310403124195cd9778ec
/avatars/57c594d3d0f97d3010b15b6a0806451c.svg
Razvan Pascanu
user
razp
661890a365e0419878fcb60b
Pier Giuseppe Sessa
661890a365e0419878fcb60c
Robert Dadashi
admin_assigned
2024-04-18T12:02:23.830Z
65fc7c0da57b88a765aea493
/avatars/440f88769da1113d6158fe7e0514ead3.svg
Robert Dadashi
user
ddsh
661890a365e0419878fcb60d
Léonard Hussenot
claimed_verified
2025-03-28T14:22:40.708Z
667a7b2e1764931987803b46
/avatars/da2ee162f33c681e3d1c4d1e6d44dcbd.svg
Léonard Hussenot
user
leonardhussenot
661890a365e0419878fcb60e
Johan Ferret
admin_assigned
2024-04-18T12:02:35.672Z
65afb7dbdd6bdfd73cd8e609
/avatars/b21069bc2d7ee4cc1508008e3c8ade64.svg
Johan Ferret
user
ferretj
661890a365e0419878fcb60f
Sertan Girgin
661890a365e0419878fcb610
Olivier Bachem
661890a365e0419878fcb611
Alek Andreev
admin_assigned
2024-04-18T12:02:54.909Z
65d667858ac5cf899e3b66c4
/avatars/340468ee8cd9125eceae23a9b8e536a5.svg
Alek Andreev
user
alekandreev
661890a365e0419878fcb612
Kathleen Kenealy
661890a365e0419878fcb613
Thomas Mesnard
661890a365e0419878fcb614
Cassidy Hardin
661890a365e0419878fcb615
Surya Bhupatiraju
admin_assigned
2024-04-18T12:03:24.765Z
65d7644d19bf1e60614dee6e
/avatars/422a8d28fa9d9ff2c3ec1406352cb346.svg
Surya Bhupatiraju
user
suryabhupa
661890a365e0419878fcb616
Shreya Pathak
661890a365e0419878fcb617
Laurent Sifre
admin_assigned
2024-04-18T12:03:35.899Z
65d74b9b7713dc8fd8956d1c
/avatars/5d5873861134bb079753c48304a7b7c2.svg
Laurent Sifre
user
laurentsifre
661890a365e0419878fcb618
Morgane Rivière
661890a365e0419878fcb619
Mihir Sanjay Kale
admin_assigned
2024-04-18T12:03:50.288Z
6226953f7551f131b4eba5c5
/avatars/eec8fa43fc37f546d9fef118a138d8a4.svg
Mihir Sanjay Kale
user
mihirkale
661890a365e0419878fcb61a
Juliette Love
661890a365e0419878fcb61b
Pouya Tafti
admin_assigned
2024-04-18T12:04:09.855Z
6558091ecafc48de36f848a8
/avatars/3b7e8327d369436d02fe8c58dde036c2.svg
Pouya Tafti
user
pdt
661890a365e0419878fcb61c
Armand Joulin
661890a365e0419878fcb61d
Noah Fiedel
admin_assigned
2024-04-18T12:04:21.963Z
62f82ebd870a3f98bbf9e62a
/avatars/3b03299e128abf2c8ba82db704855de7.svg
Noah Fiedel
user
nfiedel
661890a365e0419878fcb61e
Evan Senter
admin_assigned
2024-04-18T12:04:30.429Z
62a47ec0fdd7a13aac264904
/avatars/f2ff23645203ed2399cfe8b447af437a.svg
Evan Senter
user
evansenter
661890a365e0419878fcb61f
Yutian Chen
661890a365e0419878fcb620
Srivatsan Srinivasan
admin_assigned
2024-04-18T12:04:56.234Z
64f5121aa30d6b0a26c1b7e1
/avatars/862fe8ed200e4759835661bab161ad09.svg
Srivatsan Srinivasan
user
srivatr
661890a365e0419878fcb621
Guillaume Desjardins
661890a365e0419878fcb622
David Budden
661890a365e0419878fcb623
Arnaud Doucet
661890a365e0419878fcb624
Sharad Vikram
661890a365e0419878fcb625
Adam Paszke
661890a365e0419878fcb626
Trevor Gale
661890a365e0419878fcb627
Sebastian Borgeaud
661890a365e0419878fcb628
Charlie Chen
661890a365e0419878fcb629
Andy Brock
661890a365e0419878fcb62a
Antonia Paterson
661890a365e0419878fcb62b
Jenny Brennan
661890a365e0419878fcb62c
Meg Risdal
admin_assigned
2024-04-12T08:28:39.010Z
626b23fc822e3d85324babed
https://cdn-avatars.huggingface.co/v1/production/uploads/626b23fc822e3d85324babed/CbRuisnyy5JsCeA5uxYL7.jpeg
Meg Risdal
user
megrisdal
661890a365e0419878fcb62d
Raj Gundluru
661890a365e0419878fcb62e
Nesh Devanathan
661890a365e0419878fcb62f
Paul Mooney
admin_assigned
2024-04-18T12:05:57.471Z
644fadb9fc5e5ff919aed863
/avatars/9b646ff3eeda1f01cc2b1cb86d1634f9.svg
Paul Mooney
user
paulmooney
661890a365e0419878fcb630
Nilay Chauhan
admin_assigned
2024-04-18T12:07:38.639Z
641415cd5189abde8cbbd900
/avatars/401807f3174492ab64b0197427f21e0b.svg
Nilay Chauhan
user
nilaychauhan
661890a365e0419878fcb631
Phil Culliton
admin_assigned
2024-04-18T12:07:30.197Z
627a9aa0f967ef475abe8037
/avatars/47cf94d6c015577d0a860b617c9ea2b8.svg
Phil Culliton
user
PhilCulliton
661890a365e0419878fcb632
Luiz GUStavo Martins
admin_assigned
2024-04-18T12:07:22.567Z
637a4a31e20e8765c5220b77
/avatars/eb8e5b3bf2c1c99dad4f35ac24788669.svg
Luiz Gustavo Martins
user
gusthema
661890a365e0419878fcb633
Elisa Bandy
661890a365e0419878fcb634
David Huntsperger
admin_assigned
2024-04-18T12:07:10.781Z
65f35e8bc2d214f884656a67
/avatars/2a46a832328a1269d0077097f6a121ca.svg
David Huntsperger
user
dhuntsperger
661890a365e0419878fcb635
Glenn Cameron
661890a365e0419878fcb636
Arthur Zucker
claimed_verified
2024-04-12T14:49:59.500Z
62441cb7456803e95009a08f
https://cdn-avatars.huggingface.co/v1/production/uploads/1674683851722-62441cb7456803e95009a08f.jpeg
Arthur Zucker
user
ArthurZ
661890a365e0419878fcb637
Tris Warkentin
661890a365e0419878fcb638
Ludovic Peran
admin_assigned
2024-04-18T12:06:46.212Z
60e473dcdefbf317ecd88035
/avatars/e5c069e6cfc73ea5d3d6c2a437490cdd.svg
LUDOVIC PERAN
user
LudovicPeran
661890a365e0419878fcb639
Minh Giang
661890a365e0419878fcb63a
Zoubin Ghahramani
661890a365e0419878fcb63b
Clément Farabet
661890a365e0419878fcb63c
Koray Kavukcuoglu
661890a365e0419878fcb63d
Demis Hassabis
661890a365e0419878fcb63e
Raia Hadsell
661890a365e0419878fcb63f
Yee Whye Teh
661890a365e0419878fcb640
Nando de Frietas
RecurrentGemma: Moving Past Transformers for Efficient Open Language
  Models
We introduce RecurrentGemma, an open language model which uses Google's novel
Griffin architecture. Griffin combines linear recurrences with local attention
to achieve excellent performance on language. It has a fixed-sized state, which
reduces memory use and enables efficient inference on long sequences. We
provide a pre-trained model with 2B non-embedding parameters, and an
instruction tuned variant. Both models achieve comparable performance to
Gemma-2B despite being trained on fewer tokens.
RecurrentGemma, using Griffin architecture, achieves excellent language performance with efficient memory usage and fewer training tokens compared to Gemma-2B.
RecurrentGemma Griffin architecture linear recurrences local attention fixed-sized state efficient inference long sequences