Ultra-Long Sequence Distributed Transformer
Transformer models trained on long sequences often achieve higher accuracy
than short sequences. Unfortunately, conventional transformers struggle with
long sequence training due to the overwhelming computation and memory
requirements. Existing methods for long sequence training offer limited speedup
and memory reduction, and may compromise accuracy. This paper presents a novel
and efficient distributed training method, the Long Short-Sequence Transformer
(LSS Transformer), for training transformer with long sequences. It distributes
a long sequence into segments among GPUs, with each GPU computing a partial
self-attention for its segment. Then, it uses a fused communication and a novel
double gradient averaging technique to avoid the need to aggregate partial
self-attention and minimize communication overhead. We evaluated the
performance between LSS Transformer and the state-of-the-art Nvidia sequence
parallelism on a Wikipedia enwik8 dataset. Results show that our proposed
method lead to 5.6x faster and 10.2x more memory-efficient implementation
compared to state-of-the-art sequence parallelism on 144 Nvidia V100 GPUs.
Moreover, our algorithm scales to an extreme sequence length of 50,112 at 3,456
GPUs, achieving 161% super-linear parallel efficiency and a throughput of 32
petaflops.
6549e7c19b86bd6b2c79712e
Xiao Wang
6549e7c19b86bd6b2c79712f
Isaac Lyngaas
admin_assigned
2023-11-07T16:25:19.754Z
628cd80686fc004b14e3c7a7
/avatars/316afa1d3fd02993fe246f002627b69a.svg
Isaac Lyngaas
user
irlyngaas
6549e7c19b86bd6b2c797130
Aristeidis Tsaris
6549e7c19b86bd6b2c797131
Peng Chen
6549e7c19b86bd6b2c797132
Sajal Dash
admin_assigned
2023-11-07T16:25:50.740Z
627587897128d87e9f928d16
/avatars/fdc3e4524a5b71e9951a7ee4c3384e95.svg
Sajal Dash
user
sajaldash
6549e7c19b86bd6b2c797133
Mayanka Chandra Shekar
admin_assigned
2023-11-07T16:25:57.774Z
6283d8b0d092918e5ed99223
/avatars/6702c6a546de432b39dea7a4f1e3d706.svg
Mayanka Chandra Shekar
user
mayanka
6549e7c19b86bd6b2c797134
Tao Luo
6549e7c19b86bd6b2c797135
Hong-Jun Yoon
admin_assigned
2023-11-07T16:26:49.006Z
649edc1bd5065637167248ab
/avatars/dc6f6299c5758861b5a0f5e3ec03a419.svg
Hong-Jun Yoon
user
hongjuny
6549e7c19b86bd6b2c797136
Mohamed Wahib
6549e7c19b86bd6b2c797137
John Gouley
Ultra-Long Sequence Distributed Transformer
Transformer models trained on long sequences often achieve higher accuracy
than short sequences. Unfortunately, conventional transformers struggle with
long sequence training due to the overwhelming computation and memory
requirements. Existing methods for long sequence training offer limited speedup
and memory reduction, and may compromise accuracy. This paper presents a novel
and efficient distributed training method, the Long Short-Sequence Transformer
(LSS Transformer), for training transformer with long sequences. It distributes
a long sequence into segments among GPUs, with each GPU computing a partial
self-attention for its segment. Then, it uses a fused communication and a novel
double gradient averaging technique to avoid the need to aggregate partial
self-attention and minimize communication overhead. We evaluated the
performance between LSS Transformer and the state-of-the-art Nvidia sequence
parallelism on a Wikipedia enwik8 dataset. Results show that our proposed
method lead to 5.6x faster and 10.2x more memory-efficient implementation
compared to state-of-the-art sequence parallelism on 144 Nvidia V100 GPUs.
Moreover, our algorithm scales to an extreme sequence length of 50,112 at 3,456
GPUs, achieving 161% super-linear parallel efficiency and a throughput of 32
petaflops.
The Long Short-Sequence Transformer (LSS Transformer) efficiently trains transformers on long sequences by segmenting them across GPUs with partial self-attention and novel communication techniques, achieving significant speed and memory improvements.
Transformer models long sequences self-attention distributed training LSS Transformer sequence parallelism parallel efficiency petaflops