BurstAttention: An Efficient Distributed Attention Framework for
  Extremely Long Sequences
Effective attention modules have played a crucial role in the success of
Transformer-based large language models (LLMs), but the quadratic time and
memory complexities of these attention modules also pose a challenge when
processing long sequences. One potential solution for the long sequence problem
is to utilize distributed clusters to parallelize the computation of attention
modules across multiple devices (e.g., GPUs). However, adopting a distributed
approach inevitably introduces extra memory overheads to store local attention
results and incurs additional communication costs to aggregate local results
into global ones. In this paper, we propose a distributed attention framework
named ``BurstAttention'' to optimize memory access and communication operations
at both the global cluster and local device levels. In our experiments, we
compare BurstAttention with other competitive distributed attention solutions
for long sequence processing. The experimental results under different length
settings demonstrate that BurstAttention offers significant advantages for
processing long sequences compared with these competitive baselines, reducing
40% communication overheads and achieving 2 X speedup during training 32K
sequence length on 8 X A100.
65f3c3208e0f2653cdef4f74
Sun Ao
65f3c3208e0f2653cdef4f75
Weilin Zhao
admin_assigned
2024-03-15T13:24:24.183Z
64b8ff3d95bd42c770878042
/avatars/564a4dccdf9e5b813a99979b0ef58183.svg
Weilin Zhao
user
Achazwl
65f3c3208e0f2653cdef4f76
Xu Han
extracted_pending
2024-03-15T03:40:17.986Z
65b8bd1dd4f282b64711d049
/avatars/4e507f0ef4f3420bcff2677fd8ee14e5.svg
Xu Han
user
SillyXu
65f3c3208e0f2653cdef4f77
Cheng Yang
65f3c3208e0f2653cdef4f78
Zhiyuan Liu
admin_assigned
2024-03-15T13:25:58.934Z
6472f1f7485a7c8e1bcf507b
/avatars/79dca137583875191ea883b193b630b7.svg
Zhiyuan Liu
user
zibuyu9
65f3c3208e0f2653cdef4f79
Chuan Shi
65f3c3208e0f2653cdef4f7a
Maosong Sun
65f3c3208e0f2653cdef4f7b
Shengnan Wang
65f3c3208e0f2653cdef4f7c
Teng Su
BurstAttention: An Efficient Distributed Attention Framework for
  Extremely Long Sequences
Effective attention modules have played a crucial role in the success of
Transformer-based large language models (LLMs), but the quadratic time and
memory complexities of these attention modules also pose a challenge when
processing long sequences. One potential solution for the long sequence problem
is to utilize distributed clusters to parallelize the computation of attention
modules across multiple devices (e.g., GPUs). However, adopting a distributed
approach inevitably introduces extra memory overheads to store local attention
results and incurs additional communication costs to aggregate local results
into global ones. In this paper, we propose a distributed attention framework
named ``BurstAttention'' to optimize memory access and communication operations
at both the global cluster and local device levels. In our experiments, we
compare BurstAttention with other competitive distributed attention solutions
for long sequence processing. The experimental results under different length
settings demonstrate that BurstAttention offers significant advantages for
processing long sequences compared with these competitive baselines, reducing
40% communication overheads and achieving 2 X speedup during training 32K
sequence length on 8 X A100.
BurstAttention is a distributed attention framework that optimizes memory and communication for processing long sequences, reducing overhead and improving speed compared to existing solutions.
Transformer-based large language models attention modules quadratic time complexity quadratic memory complexity distributed clusters GPUs memory overheads communication costs distributed attention framework BurstAttention long sequence processing communication overheads speedup