DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme
  Long Sequence Transformer Models
Computation in a typical Transformer-based large language model (LLM) can be
characterized by batch size, hidden dimension, number of layers, and sequence
length. Until now, system works for accelerating LLM training have focused on
the first three dimensions: data parallelism for batch size, tensor parallelism
for hidden size and pipeline parallelism for model depth or layers. These
widely studied forms of parallelism are not targeted or optimized for long
sequence Transformer models. Given practical application needs for long
sequence LLM, renewed attentions are being drawn to sequence parallelism.
However, existing works in sequence parallelism are constrained by
memory-communication inefficiency, limiting their scalability to long sequence
large models. In this work, we introduce DeepSpeed-Ulysses, a novel, portable
and effective methodology for enabling highly efficient and scalable LLM
training with extremely long sequence length. DeepSpeed-Ulysses at its core
partitions input data along the sequence dimension and employs an efficient
all-to-all collective communication for attention computation. Theoretical
communication analysis shows that whereas other methods incur communication
overhead as sequence length increases, DeepSpeed-Ulysses maintains constant
communication volume when sequence length and compute devices are increased
proportionally. Furthermore, experimental evaluations show that
DeepSpeed-Ulysses trains 2.5X faster with 4X longer sequence length than the
existing method SOTA baseline.
6513924a606fc2019243eb7f
Sam Ade Jacobs
claimed_verified
2023-10-02T07:00:29.319Z
651736dd20b18e99b44dd5d5
/avatars/c054eb2567e027408340a03a7ca0b29d.svg
Sam Ade Jacobs
user
samadejacobs
6513924a606fc2019243eb80
Masahiro Tanaka
6513924a606fc2019243eb81
Chengming Zhang
admin_assigned
2023-09-27T10:31:09.240Z
6480355ccc9e514b3b033a78
/avatars/ff86bd27261c755c20546b400392f256.svg
Chengming Zhang
user
cc32
6513924a606fc2019243eb82
Minjia Zhang
admin_assigned
2023-09-27T10:29:23.957Z
6305b5e39d2531fabd195c5f
https://cdn-avatars.huggingface.co/v1/production/uploads/6305b5e39d2531fabd195c5f/oGS5leXVrB8Ct_BF96VkR.jpeg
Zhang
user
Minjia
6513924a606fc2019243eb83
Leon Song
claimed_verified
2023-09-27T15:21:16.142Z
651442f5c75a3d4c44de4ac9
/avatars/c60185389bf6379e4360cb54615922ed.svg
Shuaiwen Leon Song
user
leonangel991
6513924a606fc2019243eb84
Samyam Rajbhandari
admin_assigned
2023-09-27T10:29:50.197Z
63238e79d444e1d90888ea03
/avatars/cd0accb5ddec4f2ae23c94da4094b5e5.svg
Samyam Rajbhandari
user
samyam
6513924a606fc2019243eb85
Yuxiong He
DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme
  Long Sequence Transformer Models
Computation in a typical Transformer-based large language model (LLM) can be
characterized by batch size, hidden dimension, number of layers, and sequence
length. Until now, system works for accelerating LLM training have focused on
the first three dimensions: data parallelism for batch size, tensor parallelism
for hidden size and pipeline parallelism for model depth or layers. These
widely studied forms of parallelism are not targeted or optimized for long
sequence Transformer models. Given practical application needs for long
sequence LLM, renewed attentions are being drawn to sequence parallelism.
However, existing works in sequence parallelism are constrained by
memory-communication inefficiency, limiting their scalability to long sequence
large models. In this work, we introduce DeepSpeed-Ulysses, a novel, portable
and effective methodology for enabling highly efficient and scalable LLM
training with extremely long sequence length. DeepSpeed-Ulysses at its core
partitions input data along the sequence dimension and employs an efficient
all-to-all collective communication for attention computation. Theoretical
communication analysis shows that whereas other methods incur communication
overhead as sequence length increases, DeepSpeed-Ulysses maintains constant
communication volume when sequence length and compute devices are increased
proportionally. Furthermore, experimental evaluations show that
DeepSpeed-Ulysses trains 2.5X faster with 4X longer sequence length than the
existing method SOTA baseline.
DeepSpeed-Ulysses enables efficient and scalable training of Transformer-based LLMs with long sequences by partitioning input data along the sequence dimension and using all-to-all communication.
Transformer-based large language model LLM data parallelism tensor parallelism pipeline parallelism sequence parallelism memory-communication inefficiency DeepSpeed-Ulysses all-to-all collective communication attention computation communication analysis