Eager Updates For Overlapped Communication and Computation in DiLoCo
Distributed optimization methods such as DiLoCo have been shown to be
effective in training very large models across multiple distributed workers,
such as datacenters. These methods split updates into two parts: an inner
optimization phase, where the workers independently execute multiple
optimization steps on their own local data, and an outer optimization step,
where the inner updates are synchronized. While such approaches require orders
of magnitude less communication than standard data-parallel training, in
settings where the workers are datacenters, even the limited communication
requirements of these approaches can still cause significant slow downs due to
the blocking necessary at each outer optimization step. In this paper, we
investigate techniques to mitigate this issue by overlapping communication with
computation in a manner that allows the outer optimization step to fully
overlap with the inner optimization phase. We show that a particular variant,
dubbed eager updates, provides competitive performance with standard DiLoCo in
settings with low bandwidth between workers.
67b5bcd091132877cf330179
Satyen Kale
67b5bcd091132877cf33017a
Arthur Douillard
admin_assigned
2025-02-19T14:42:51.668Z
622792366303bf1dc304f49f
/avatars/975c1cc3eb2f97cf8e848162056d5bea.svg
Arthur Douillard
user
ArthurDouillard
67b5bcd091132877cf33017b
Yanislav Donchev
admin_assigned
2025-02-19T14:42:57.488Z
61796df92357d28cdf952511
/avatars/01821f364cf78c95aed1986d91d40610.svg
Yanislav Donchev
user
yannidd
Eager Updates For Overlapped Communication and Computation in DiLoCo
Distributed optimization methods such as DiLoCo have been shown to be
effective in training very large models across multiple distributed workers,
such as datacenters. These methods split updates into two parts: an inner
optimization phase, where the workers independently execute multiple
optimization steps on their own local data, and an outer optimization step,
where the inner updates are synchronized. While such approaches require orders
of magnitude less communication than standard data-parallel training, in
settings where the workers are datacenters, even the limited communication
requirements of these approaches can still cause significant slow downs due to
the blocking necessary at each outer optimization step. In this paper, we
investigate techniques to mitigate this issue by overlapping communication with
computation in a manner that allows the outer optimization step to fully
overlap with the inner optimization phase. We show that a particular variant,
dubbed eager updates, provides competitive performance with standard DiLoCo in
settings with low bandwidth between workers.
Techniques to overlap communication with computation in distributed optimization methods like DiLoCo reduce synchronization delays in distributed training environments.
DiLoCo distributed optimization inner optimization phase outer optimization step eager updates