Streaming DiLoCo with overlapping communication: Towards a Distributed
  Free Lunch
Training of large language models (LLMs) is typically distributed across a
large number of accelerators to reduce training time. Since internal states and
parameter gradients need to be exchanged at each and every single gradient
step, all devices need to be co-located using low-latency high-bandwidth
communication links to support the required high volume of exchanged bits.
Recently, distributed algorithms like DiLoCo have relaxed such co-location
constraint: accelerators can be grouped into ``workers'', where
synchronizations between workers only occur infrequently. This in turn means
that workers can afford being connected by lower bandwidth communication links
without affecting learning quality. However, in these methods, communication
across workers still requires the same peak bandwidth as before, as the
synchronizations require all parameters to be exchanged across all workers. In
this paper, we improve DiLoCo in three ways. First, we synchronize only subsets
of parameters in sequence, rather than all at once, which greatly reduces peak
bandwidth. Second, we allow workers to continue training while synchronizing,
which decreases wall clock time. Third, we quantize the data exchanged by
workers, which further reduces bandwidth across workers. By properly combining
these modifications, we show experimentally that we can distribute training of
billion-scale parameters and reach similar quality as before, but reducing
required bandwidth by two orders of magnitude.
679ca01ecad2402cec0a939a
Arthur Douillard
679ca01ecad2402cec0a939b
Yanislav Donchev
679ca01ecad2402cec0a939c
Keith Rush
claimed_verified
2025-03-16T21:21:00.258Z
65f1cf3aea7a16c6295f4737
/avatars/905106897916e87e114ccc2d33e8e1c2.svg
Keith Rush
user
jkr26
679ca01ecad2402cec0a939d
Satyen Kale
679ca01ecad2402cec0a939e
Zachary Charles
679ca01ecad2402cec0a939f
Zachary Garrett
claimed_verified
2025-03-16T21:20:51.929Z
65f1be9109a3517bd1c8e17b
/avatars/bc4112a3f749fe969ebb633b2e0cbc14.svg
Zachary Garrett
user
ZacharyGarrett
679ca01ecad2402cec0a93a0
Gabriel Teston
claimed_verified
2025-03-16T21:20:57.817Z
63177ee2879073cf83e8b2ab
/avatars/e1b7974fd48c273616c13390a8052bf0.svg
Gabriel Teston
user
Gabriel-Teston
679ca01ecad2402cec0a93a1
Dave Lacey
679ca01ecad2402cec0a93a2
Ross McIlroy
679ca01ecad2402cec0a93a3
Jiajun Shen
679ca01ecad2402cec0a93a4
Alexandre Ramé
679ca01ecad2402cec0a93a5
Arthur Szlam
679ca01ecad2402cec0a93a6
Marc'Aurelio Ranzato
679ca01ecad2402cec0a93a7
Paul Barham
Streaming DiLoCo with overlapping communication: Towards a Distributed
  Free Lunch
Training of large language models (LLMs) is typically distributed across a
large number of accelerators to reduce training time. Since internal states and
parameter gradients need to be exchanged at each and every single gradient
step, all devices need to be co-located using low-latency high-bandwidth
communication links to support the required high volume of exchanged bits.
Recently, distributed algorithms like DiLoCo have relaxed such co-location
constraint: accelerators can be grouped into ``workers'', where
synchronizations between workers only occur infrequently. This in turn means
that workers can afford being connected by lower bandwidth communication links
without affecting learning quality. However, in these methods, communication
across workers still requires the same peak bandwidth as before, as the
synchronizations require all parameters to be exchanged across all workers. In
this paper, we improve DiLoCo in three ways. First, we synchronize only subsets
of parameters in sequence, rather than all at once, which greatly reduces peak
bandwidth. Second, we allow workers to continue training while synchronizing,
which decreases wall clock time. Third, we quantize the data exchanged by
workers, which further reduces bandwidth across workers. By properly combining
these modifications, we show experimentally that we can distribute training of
billion-scale parameters and reach similar quality as before, but reducing
required bandwidth by two orders of magnitude.
The paper presents enhancements to DiLoCo for distributed training of large language models, reducing peak bandwidth and wall clock time while maintaining learning quality.
large language models (LLMs) distributed training accelerators low-latency communication high-bandwidth communication DiLoCo parameter synchronization peak bandwidth wall clock time quantization