LongNet: Scaling Transformers to 1,000,000,000 Tokens
Scaling sequence length has become a critical demand in the era of large
language models. However, existing methods struggle with either computational
complexity or model expressivity, rendering the maximum sequence length
restricted. In this work, we introduce LongNet, a Transformer variant that can
scale sequence length to more than 1 billion tokens, without sacrificing the
performance on shorter sequences. Specifically, we propose dilated attention,
which expands the attentive field exponentially as the distance grows. LongNet
has significant advantages: 1) it has a linear computation complexity and a
logarithm dependency between tokens; 2) it can be served as a distributed
trainer for extremely long sequences; 3) its dilated attention is a drop-in
replacement for standard attention, which can be seamlessly integrated with the
existing Transformer-based optimization. Experiments results demonstrate that
LongNet yields strong performance on both long-sequence modeling and general
language tasks. Our work opens up new possibilities for modeling very long
sequences, e.g., treating a whole corpus or even the entire Internet as a
sequence.
64a6234ba3227ff52076ea25
Jiayu Ding
64a6234ba3227ff52076ea26
Shuming Ma
64a6234ba3227ff52076ea27
Li Dong
claimed_verified
2024-10-08T15:20:01.888Z
5df85abada6d0311fd3d5408
/avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
Li Dong
user
unilm
64a6234ba3227ff52076ea28
Xingxing Zhang
64a6234ba3227ff52076ea29
Shaohan Huang
admin_assigned
2023-07-06T12:15:34.559Z
632bd2f72d6a805eeb4bc601
/avatars/6e1533e8a599f3068290aa69ac82cab7.svg
HUANG SHAOHAN
user
buaahsh
64a6234ba3227ff52076ea2a
Wenhui Wang
64a6234ba3227ff52076ea2b
Furu Wei
admin_assigned
2023-07-06T12:17:15.965Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
LongNet: Scaling Transformers to 1,000,000,000 Tokens
Scaling sequence length has become a critical demand in the era of large
language models. However, existing methods struggle with either computational
complexity or model expressivity, rendering the maximum sequence length
restricted. In this work, we introduce LongNet, a Transformer variant that can
scale sequence length to more than 1 billion tokens, without sacrificing the
performance on shorter sequences. Specifically, we propose dilated attention,
which expands the attentive field exponentially as the distance grows. LongNet
has significant advantages: 1) it has a linear computation complexity and a
logarithm dependency between tokens; 2) it can be served as a distributed
trainer for extremely long sequences; 3) its dilated attention is a drop-in
replacement for standard attention, which can be seamlessly integrated with the
existing Transformer-based optimization. Experiments results demonstrate that
LongNet yields strong performance on both long-sequence modeling and general
language tasks. Our work opens up new possibilities for modeling very long
sequences, e.g., treating a whole corpus or even the entire Internet as a
sequence.
LongNet, a Transformer variant with dilated attention, enables scaling to over 1 billion tokens with linear computational complexity and maintains performance on shorter sequences.
Transformer dilated attention sequence length computational complexity long-sequence modeling general language tasks