DiLoCoX: A Low-Communication Large-Scale Training Framework for
  Decentralized Cluster
The distributed training of foundation models, particularly large language
models (LLMs), demands a high level of communication. Consequently, it is
highly dependent on a centralized cluster with fast and reliable interconnects.
Can we conduct training on slow networks and thereby unleash the power of
decentralized clusters when dealing with models exceeding 100 billion
parameters? In this paper, we propose DiLoCoX, a low-communication large-scale
decentralized cluster training framework. It combines Pipeline Parallelism with
Dual Optimizer Policy, One-Step-Delay Overlap of Communication and Local
Training, and an Adaptive Gradient Compression Scheme. This combination
significantly improves the scale of parameters and the speed of model
pre-training. We justify the benefits of one-step-delay overlap of
communication and local training, as well as the adaptive gradient compression
scheme, through a theoretical analysis of convergence. Empirically, we
demonstrate that DiLoCoX is capable of pre-training a 107B foundation model
over a 1Gbps network. Compared to vanilla AllReduce, DiLoCoX can achieve a 357x
speedup in distributed training while maintaining negligible degradation in
model convergence. To the best of our knowledge, this is the first
decentralized training framework successfully applied to models with over 100
billion parameters.
685e8cf171131fa43be08bb5
Ji Qi
685e8cf171131fa43be08bb6
WenPeng Zhu
685e8cf171131fa43be08bb7
Li Li
685e8cf171131fa43be08bb8
Ming Wu
685e8cf171131fa43be08bb9
YingJun Wu
685e8cf171131fa43be08bba
Wu He
685e8cf171131fa43be08bbb
Xun Gao
685e8cf171131fa43be08bbc
Jason Zeng
685e8cf171131fa43be08bbd
Michael Heinrich
DiLoCoX: A Low-Communication Large-Scale Training Framework for
  Decentralized Cluster
The distributed training of foundation models, particularly large language
models (LLMs), demands a high level of communication. Consequently, it is
highly dependent on a centralized cluster with fast and reliable interconnects.
Can we conduct training on slow networks and thereby unleash the power of
decentralized clusters when dealing with models exceeding 100 billion
parameters? In this paper, we propose DiLoCoX, a low-communication large-scale
decentralized cluster training framework. It combines Pipeline Parallelism with
Dual Optimizer Policy, One-Step-Delay Overlap of Communication and Local
Training, and an Adaptive Gradient Compression Scheme. This combination
significantly improves the scale of parameters and the speed of model
pre-training. We justify the benefits of one-step-delay overlap of
communication and local training, as well as the adaptive gradient compression
scheme, through a theoretical analysis of convergence. Empirically, we
demonstrate that DiLoCoX is capable of pre-training a 107B foundation model
over a 1Gbps network. Compared to vanilla AllReduce, DiLoCoX can achieve a 357x
speedup in distributed training while maintaining negligible degradation in
model convergence. To the best of our knowledge, this is the first
decentralized training framework successfully applied to models with over 100
billion parameters.
DiLoCoX, a decentralized cluster training framework, enhances the training of large-scale models over slow networks by utilizing pipeline parallelism, dual optimizer policy, and gradient compression, achieving significant speed improvements and effective scalability.
Pipeline Parallelism Dual Optimizer Policy One-Step-Delay Overlap Adaptive Gradient Compression Scheme decentralized training large-scale models gradient compression model convergence AllReduce