ZClip: Adaptive Spike Mitigation for LLM Pre-Training
Training large language models (LLMs) presents numerous challenges, including
gradient instability and loss spikes. These phenomena can lead to catastrophic
divergence, requiring costly checkpoint restoration and data batch skipping.
Traditional gradient clipping techniques, such as constant or norm-based
methods, fail to address these issues effectively due to their reliance on
fixed thresholds or heuristics, leading to inefficient learning and requiring
frequent manual intervention. In this work, we propose ZClip, an adaptive
gradient clipping algorithm that dynamically adjusts the clipping threshold
based on statistical properties of gradient norms over time. Unlike prior
reactive strategies, ZClip proactively adapts to training dynamics without
making any prior assumptions on the scale and the temporal evolution of
gradient norms. At its core, it leverages z-score-based anomaly detection to
identify and mitigate large gradient spikes, preventing malignant loss spikes
while not interfering with convergence otherwise. Our code is available at:
https://github.com/bluorion-com/ZClip.
67ef5a3d4417508df8d99dad
Abhay Kumar
claimed_verified
2025-04-04T07:08:18.512Z
62cd4b03c5cc157be82f0b56
/avatars/351e963c1c763d507ae78cbcd62966a3.svg
Abhay kumar
user
akanyaani
67ef5a3d4417508df8d99dae
Louis Owen
claimed_verified
2025-04-04T07:08:21.051Z
6071c4b270e11b30cfcfd7a3
https://cdn-avatars.huggingface.co/v1/production/uploads/6071c4b270e11b30cfcfd7a3/-1ekCBzSTpqxkkul0bgmI.jpeg
Louis Owen
user
louisowen6
67ef5a3d4417508df8d99daf
Nilabhra Roy Chowdhury
claimed_verified
2025-04-04T07:08:12.764Z
645a0d3dd6648853107c5fdc
/avatars/1e3b6a4f5ce81a707ba7cbdf81631091.svg
Nilabhra Roy Chowdhury
user
nilabhra
67ef5a3d4417508df8d99db0
Fabian Güra
claimed_verified
2025-04-04T07:08:16.132Z
65e4be59e8b017ee1310a1b6
/avatars/c3f7cdf5d0859cb80bfb2b970a675dfa.svg
Fabian
user
gueraf
ZClip: Adaptive Spike Mitigation for LLM Pre-Training
Training large language models (LLMs) presents numerous challenges, including
gradient instability and loss spikes. These phenomena can lead to catastrophic
divergence, requiring costly checkpoint restoration and data batch skipping.
Traditional gradient clipping techniques, such as constant or norm-based
methods, fail to address these issues effectively due to their reliance on
fixed thresholds or heuristics, leading to inefficient learning and requiring
frequent manual intervention. In this work, we propose ZClip, an adaptive
gradient clipping algorithm that dynamically adjusts the clipping threshold
based on statistical properties of gradient norms over time. Unlike prior
reactive strategies, ZClip proactively adapts to training dynamics without
making any prior assumptions on the scale and the temporal evolution of
gradient norms. At its core, it leverages z-score-based anomaly detection to
identify and mitigate large gradient spikes, preventing malignant loss spikes
while not interfering with convergence otherwise. Our code is available at:
https://github.com/bluorion-com/ZClip.
ZClip is an adaptive gradient clipping algorithm that uses z-score-based anomaly detection to dynamically adjust clipping thresholds and prevent large gradient spikes during LLM training.
gradient instability loss spikes catastrophic divergence gradient clipping adaptive gradient clipping z-score-based anomaly detection