RWKV: Reinventing RNNs for the Transformer Era
Transformers have revolutionized almost all natural language processing (NLP)
tasks but suffer from memory and computational complexity that scales
quadratically with sequence length. In contrast, recurrent neural networks
(RNNs) exhibit linear scaling in memory and computational requirements but
struggle to match the same performance as Transformers due to limitations in
parallelization and scalability. We propose a novel model architecture,
Receptance Weighted Key Value (RWKV), that combines the efficient
parallelizable training of Transformers with the efficient inference of RNNs.
Our approach leverages a linear attention mechanism and allows us to formulate
the model as either a Transformer or an RNN, which parallelizes computations
during training and maintains constant computational and memory complexity
during inference, leading to the first non-transformer architecture to be
scaled to tens of billions of parameters. Our experiments reveal that RWKV
performs on par with similarly sized Transformers, suggesting that future work
can leverage this architecture to create more efficient models. This work
presents a significant step towards reconciling the trade-offs between
computational efficiency and model performance in sequence processing tasks.
646c131ddf2609a541c4a1cc
Bo Peng
646c131ddf2609a541c4a1cd
Eric Alcaide
claimed_verified
2023-05-23T10:23:58.453Z
646c8a411ee398a4e940cab1
https://cdn-avatars.huggingface.co/v1/production/uploads/646c8a411ee398a4e940cab1/5W3dGj8eQqQrxeGPQO1n2.png
hypnopump
user
hypnopump
646c131ddf2609a541c4a1ce
Quentin Anthony
claimed_verified
2023-05-24T07:51:26.034Z
63b9a261efe99543b34e9579
https://cdn-avatars.huggingface.co/v1/production/uploads/63b9a261efe99543b34e9579/aZMhhnurORuPU_WryDUiu.jpeg
Quentin Anthony
user
qanthony
646c131ddf2609a541c4a1cf
Alon Albalak
claimed_verified
2023-05-24T07:51:23.306Z
611a7ec4289467cafea62d13
https://cdn-avatars.huggingface.co/v1/production/uploads/611a7ec4289467cafea62d13/pck-0fmPQkoU7yzh6-WoL.jpeg
Alon Albalak
user
alon-albalak
646c131ddf2609a541c4a1d0
Samuel Arcadinho
claimed_verified
2024-10-11T10:08:40.371Z
60eeedbf50b60c406afc1291
https://cdn-avatars.huggingface.co/v1/production/uploads/1649111275459-60eeedbf50b60c406afc1291.png
Samuel Arcadinho
user
SSamDav
646c131ddf2609a541c4a1d1
Huanqi Cao
646c131ddf2609a541c4a1d2
Xin Cheng
646c131ddf2609a541c4a1d3
Michael Chung
646c131ddf2609a541c4a1d4
Matteo Grella
646c131ddf2609a541c4a1d5
Kranthi Kiran GV
646c131ddf2609a541c4a1d6
Xuzheng He
646c131ddf2609a541c4a1d7
Haowen Hou
claimed_verified
2025-03-20T19:17:58.467Z
64414d0a66a62c605c94d14d
https://cdn-avatars.huggingface.co/v1/production/uploads/64414d0a66a62c605c94d14d/KGjlf9tRwslr7Q503YLnJ.jpeg
howard-hou
user
howard-hou
646c131ddf2609a541c4a1d8
Przemyslaw Kazienko
646c131ddf2609a541c4a1d9
Jan Kocon
646c131ddf2609a541c4a1da
Jiaming Kong
646c131ddf2609a541c4a1db
Bartlomiej Koptyra
646c131ddf2609a541c4a1dc
Hayden Lau
646c131ddf2609a541c4a1dd
Krishna Sri Ipsit Mantri
646c131ddf2609a541c4a1de
Ferdinand Mom
claimed_verified
2023-05-23T08:11:07.996Z
64622c4093f702673bf9b953
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/iB42uASF1DoQth23cizSh.png
Ferdinand Mom
user
3outeille
646c131ddf2609a541c4a1df
Atsushi Saito
claimed_verified
2024-04-17T13:15:20.653Z
606b1d984b47ca0dcae61aed
/avatars/2457293c5469f2bc5116f6e915f045c8.svg
AtsushiSaito
user
atsushi3110
646c131ddf2609a541c4a1e0
Xiangru Tang
claimed_verified
2024-02-07T11:17:39.982Z
63357c608adfa81faf2ac180
/avatars/ae0314c644f882251baf59b9134fd36f.svg
Xiangru Tang
user
RTT1
646c131ddf2609a541c4a1e1
Bolun Wang
646c131ddf2609a541c4a1e2
Johan S. Wind
646c131ddf2609a541c4a1e3
Stansilaw Wozniak
646c131ddf2609a541c4a1e4
Ruichong Zhang
claimed_verified
2023-05-23T06:39:23.050Z
6418629fd13ffa408128d7ae
https://cdn-avatars.huggingface.co/v1/production/uploads/1679319546731-noauth.png
Zhang Ruichong
user
ZhangRC
646c131ddf2609a541c4a1e5
Zhenyuan Zhang
646c131ddf2609a541c4a1e6
Qihang Zhao
646c131ddf2609a541c4a1e7
Peng Zhou
646c131ddf2609a541c4a1e8
Jian Zhu
646c131ddf2609a541c4a1e9
Rui-Jie Zhu
claimed_verified
2024-10-14T19:08:33.084Z
63ff09f24852102d4871c19c
https://cdn-avatars.huggingface.co/v1/production/uploads/63ff09f24852102d4871c19c/lyE3xemtZss3qebK5sEXw.png
Rui-Jie Zhu
user
ridger
RWKV: Reinventing RNNs for the Transformer Era
Transformers have revolutionized almost all natural language processing (NLP)
tasks but suffer from memory and computational complexity that scales
quadratically with sequence length. In contrast, recurrent neural networks
(RNNs) exhibit linear scaling in memory and computational requirements but
struggle to match the same performance as Transformers due to limitations in
parallelization and scalability. We propose a novel model architecture,
Receptance Weighted Key Value (RWKV), that combines the efficient
parallelizable training of Transformers with the efficient inference of RNNs.
Our approach leverages a linear attention mechanism and allows us to formulate
the model as either a Transformer or an RNN, which parallelizes computations
during training and maintains constant computational and memory complexity
during inference, leading to the first non-transformer architecture to be
scaled to tens of billions of parameters. Our experiments reveal that RWKV
performs on par with similarly sized Transformers, suggesting that future work
can leverage this architecture to create more efficient models. This work
presents a significant step towards reconciling the trade-offs between
computational efficiency and model performance in sequence processing tasks.
A new architecture, RWKV, combines the parallelizable training of Transformers with the efficient inference of RNNs, achieving linear scaling and matching performance.
Transformers recurrent neural networks (RNNs) linear attention mechanism parallelizable training inference RWKV