All is Not Lost: LLM Recovery without Checkpoints
Training LLMs on decentralized and wimpy computation nodes, e.g., multiple
on-spot instances, lowers the training cost and enables model democratization.
The inevitable challenge here is the churn of nodes due to failures and the
operator's scheduling policies, leading to losing a stage - a part of the
model. The conventional approaches to recover from failures are to either use
checkpointing, where periodically a copy of the entire model is sent to an
additional storage, or redundant computation. These approaches yield
significant communication and/or computation overhead even in non-failure cases
and scale poorly in settings with large models. In this paper, we propose,
CheckFree, an efficient recovery method where a failing stage is substituted by
a weighted average of the closest neighboring stages. In contrast to the state
of the art, CheckFree requires no additional computation or storage. However,
because of the nature of averaging neighbouring stages, it can only recover
failures of intermediate stages. We further extend our method to CheckFree+
with out-of-order pipeline execution to tolerate crashes of the first and last
stages. Thanks to out-of-order pipelining, behaviour of those stages is
mimicked by their neighboring ones, which allows CheckFree+ to recover them by
simply copying the weights from the immediate neighbour. To be able to recover
the (de)embedding layers, CheckFree+ copies those layers to the neighboring
stages, which requires relatively small storage overhead. We extensively
evaluate our method on LLaMa models of model sizes from 124M to 1.5B with
varying failure frequencies. In the case of low and medium failure rates
(5-10%), CheckFree and CheckFree+ outperform both checkpointing and redundant
computation in terms of convergence in wall-clock time by over 12%. Both of our
proposals can be run via our code available at:
https://github.com/gensyn-ai/CheckFree.
68546c107bc8d012d4ca9936
Nikolay Blagoev
admin_assigned
2025-06-20T12:29:18.612Z
66e17afa21dfd93e62ff5433
/avatars/5962116b5198e7ec7c18f4d10982265b.svg
Nikolay Blagoev
user
Faustify
68546c107bc8d012d4ca9937
Oğuzhan Ersoy
claimed_verified
2025-06-20T11:02:16.265Z
673c712c8a9ba30a57741a8c
https://cdn-avatars.huggingface.co/v1/production/uploads/673c712c8a9ba30a57741a8c/khBSYmO5V1BM8Q_uPLLWB.jpeg
Oguzhan Ersoy
user
oguzer
68546c107bc8d012d4ca9938
Lydia Yiyu Chen
All is Not Lost: LLM Recovery without Checkpoints
Training LLMs on decentralized and wimpy computation nodes, e.g., multiple
on-spot instances, lowers the training cost and enables model democratization.
The inevitable challenge here is the churn of nodes due to failures and the
operator's scheduling policies, leading to losing a stage - a part of the
model. The conventional approaches to recover from failures are to either use
checkpointing, where periodically a copy of the entire model is sent to an
additional storage, or redundant computation. These approaches yield
significant communication and/or computation overhead even in non-failure cases
and scale poorly in settings with large models. In this paper, we propose,
CheckFree, an efficient recovery method where a failing stage is substituted by
a weighted average of the closest neighboring stages. In contrast to the state
of the art, CheckFree requires no additional computation or storage. However,
because of the nature of averaging neighbouring stages, it can only recover
failures of intermediate stages. We further extend our method to CheckFree+
with out-of-order pipeline execution to tolerate crashes of the first and last
stages. Thanks to out-of-order pipelining, behaviour of those stages is
mimicked by their neighboring ones, which allows CheckFree+ to recover them by
simply copying the weights from the immediate neighbour. To be able to recover
the (de)embedding layers, CheckFree+ copies those layers to the neighboring
stages, which requires relatively small storage overhead. We extensively
evaluate our method on LLaMa models of model sizes from 124M to 1.5B with
varying failure frequencies. In the case of low and medium failure rates
(5-10%), CheckFree and CheckFree+ outperform both checkpointing and redundant
computation in terms of convergence in wall-clock time by over 12%. Both of our
proposals can be run via our code available at:
https://github.com/gensyn-ai/CheckFree.
A novel method, CheckFree, and its extended version CheckFree+, efficiently recover from node failures during LLM training by substituting failed stages with averaged neighboring stages or through out-of-order pipeline execution, improving convergence time over existing checkpointing methods.
LLMs decentralized computation node churn checkpointing redundant computation out-of-order pipeline execution LLaMa models (de)embedding layers wall-clock time convergence