Retentive Network: A Successor to Transformer for Large Language Models
In this work, we propose Retentive Network (RetNet) as a foundation
architecture for large language models, simultaneously achieving training
parallelism, low-cost inference, and good performance. We theoretically derive
the connection between recurrence and attention. Then we propose the retention
mechanism for sequence modeling, which supports three computation paradigms,
i.e., parallel, recurrent, and chunkwise recurrent. Specifically, the parallel
representation allows for training parallelism. The recurrent representation
enables low-cost O(1) inference, which improves decoding throughput, latency,
and GPU memory without sacrificing performance. The chunkwise recurrent
representation facilitates efficient long-sequence modeling with linear
complexity, where each chunk is encoded parallelly while recurrently
summarizing the chunks. Experimental results on language modeling show that
RetNet achieves favorable scaling results, parallel training, low-cost
deployment, and efficient inference. The intriguing properties make RetNet a
strong successor to Transformer for large language models. Code will be
available at https://aka.ms/retnet.
64b5fb597d9d8f9970f9f7e0
Yutao Sun
admin_assigned
2023-07-18T07:43:44.054Z
6300ef4779c5ddbc6cf83e1a
https://cdn-avatars.huggingface.co/v1/production/uploads/1661005591657-noauth.jpeg
Yutao Sun
user
sunyt32
64b5fb597d9d8f9970f9f7e1
Li Dong
admin_assigned
2024-10-16T09:49:49.350Z
5df85abada6d0311fd3d5408
/avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
Li Dong
user
unilm
64b5fb597d9d8f9970f9f7e2
Shaohan Huang
admin_assigned
2023-07-18T07:44:52.750Z
632bd2f72d6a805eeb4bc601
/avatars/6e1533e8a599f3068290aa69ac82cab7.svg
HUANG SHAOHAN
user
buaahsh
64b5fb597d9d8f9970f9f7e3
Shuming Ma
admin_assigned
2023-07-18T07:45:10.079Z
613f07f40153aafa379775f2
/avatars/3965175b320d753d9a5ccb0c7d9298a4.svg
Shuming Ma
user
shumingma
64b5fb597d9d8f9970f9f7e4
Yuqing Xia
admin_assigned
2023-07-18T07:45:25.152Z
6369e01864aad59d4d4501ac
/avatars/bcbd3f9d0d194eeccd061c4fa6a6e283.svg
Yuqing Xia
user
yuqxia
64b5fb597d9d8f9970f9f7e5
Jilong Xue
admin_assigned
2023-07-18T07:45:42.578Z
632bfc0e4a4991e7114f2a83
/avatars/19f8154a30d4787c5ebdf9657e37ea12.svg
Jilong Xue
user
jlxue
64b5fb597d9d8f9970f9f7e6
Jianyong Wang
64b5fb597d9d8f9970f9f7e7
Furu Wei
admin_assigned
2023-07-18T07:46:25.505Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
Retentive Network: A Successor to Transformer for Large Language Models
In this work, we propose Retentive Network (RetNet) as a foundation
architecture for large language models, simultaneously achieving training
parallelism, low-cost inference, and good performance. We theoretically derive
the connection between recurrence and attention. Then we propose the retention
mechanism for sequence modeling, which supports three computation paradigms,
i.e., parallel, recurrent, and chunkwise recurrent. Specifically, the parallel
representation allows for training parallelism. The recurrent representation
enables low-cost O(1) inference, which improves decoding throughput, latency,
and GPU memory without sacrificing performance. The chunkwise recurrent
representation facilitates efficient long-sequence modeling with linear
complexity, where each chunk is encoded parallelly while recurrently
summarizing the chunks. Experimental results on language modeling show that
RetNet achieves favorable scaling results, parallel training, low-cost
deployment, and efficient inference. The intriguing properties make RetNet a
strong successor to Transformer for large language models. Code will be
available at https://aka.ms/retnet.
Retentive Network (RetNet) offers a foundation for large language models with parallel training, low-cost inference, and efficient long-sequence modeling through a retention mechanism supporting parallel, recurrent, and chunkwise recurrent paradigms.
Retentive Network RetNet sequence modeling parallel representation recurrent representation chunkwise recurrent representation Transformer language modeling scaling results parallel training low-cost deployment efficient inference