Stable-SPAM: How to Train in 4-Bit More Stably than 16-Bit Adam
This paper comprehensively evaluates several recently proposed optimizers for
4-bit training, revealing that low-bit precision amplifies sensitivity to
learning rates and often causes unstable gradient norms, leading to divergence
at higher learning rates. Among these, SPAM, a recent optimizer featuring
momentum reset and spike-aware gradient clipping, achieves the best performance
across various bit levels, but struggles to stabilize gradient norms, requiring
careful learning rate tuning. To address these limitations, we propose
Stable-SPAM, which incorporates enhanced gradient normalization and clipping
techniques. In particular, Stable-SPAM (1) adaptively updates the clipping
threshold for spiked gradients by tracking their historical maxima; (2)
normalizes the entire gradient matrix based on its historical l_2-norm
statistics; and (3) inherits momentum reset from SPAM to periodically reset
the first and second moments of Adam, mitigating the accumulation of spiked
gradients. Extensive experiments show that Stable-SPAM effectively stabilizes
gradient norms in 4-bit LLM training, delivering superior performance compared
to Adam and SPAM. Notably, our 4-bit LLaMA-1B model trained with Stable-SPAM
outperforms the BF16 LLaMA-1B trained with Adam by up to 2 perplexity.
Furthermore, when both models are trained in 4-bit, Stable-SPAM achieves the
same loss as Adam while requiring only about half the training steps. Code is
available at https://github.com/TianjinYellow/StableSPAM.git.
67bd9b40478ef7c36240c6e6
Tianjin Huang
admin_assigned
2025-02-25T16:30:14.371Z
64cd4743a785f2043b32915e
/avatars/ba0b497a194dfea8449112d71fc67654.svg
Tianjin Huang
user
TianjinHuang
67bd9b40478ef7c36240c6e7
Haotian Hu
claimed_verified
2025-02-26T08:30:36.869Z
67577b39a6073ee33f97cdd9
/avatars/c05788ebc7c59b13fcddf8ff88540f79.svg
Haotian Hu
user
cspikachu
67bd9b40478ef7c36240c6e8
Zhenyu Zhang
claimed_verified
2025-02-26T08:30:38.930Z
649c888f67fd6c6aa97e5f85
/avatars/9967b729916d1128773102797fed1673.svg
Zhenyu Zhang
user
Kyriection
67bd9b40478ef7c36240c6e9
Gaojie Jin
admin_assigned
2025-02-25T16:30:59.295Z
6662bd6b6016f6effa6ce492
/avatars/c8e6b4b4a64f7ca0ddc91af5217a791b.svg
gaojie jin
user
sggjin
67bd9b40478ef7c36240c6ea
Xiang Li
67bd9b40478ef7c36240c6eb
Li Shen
67bd9b40478ef7c36240c6ec
Tianlong Chen
67bd9b40478ef7c36240c6ed
Lu Liu
67bd9b40478ef7c36240c6ee
Qingsong Wen
67bd9b40478ef7c36240c6ef
Zhangyang Wang
67bd9b40478ef7c36240c6f0
Shiwei Liu
claimed_verified
2025-03-03T11:14:48.533Z
65b04d2291e63920a7898c9e
https://cdn-avatars.huggingface.co/v1/production/uploads/65b04d2291e63920a7898c9e/iUHs235G4bqK-KnH_94ti.jpeg
Liu
user
Shiweiliuiiiiiii
Stable-SPAM: How to Train in 4-Bit More Stably than 16-Bit Adam
This paper comprehensively evaluates several recently proposed optimizers for
4-bit training, revealing that low-bit precision amplifies sensitivity to
learning rates and often causes unstable gradient norms, leading to divergence
at higher learning rates. Among these, SPAM, a recent optimizer featuring
momentum reset and spike-aware gradient clipping, achieves the best performance
across various bit levels, but struggles to stabilize gradient norms, requiring
careful learning rate tuning. To address these limitations, we propose
Stable-SPAM, which incorporates enhanced gradient normalization and clipping
techniques. In particular, Stable-SPAM (1) adaptively updates the clipping
threshold for spiked gradients by tracking their historical maxima; (2)
normalizes the entire gradient matrix based on its historical l_2-norm
statistics; and (3) inherits momentum reset from SPAM to periodically reset
the first and second moments of Adam, mitigating the accumulation of spiked
gradients. Extensive experiments show that Stable-SPAM effectively stabilizes
gradient norms in 4-bit LLM training, delivering superior performance compared
to Adam and SPAM. Notably, our 4-bit LLaMA-1B model trained with Stable-SPAM
outperforms the BF16 LLaMA-1B trained with Adam by up to 2 perplexity.
Furthermore, when both models are trained in 4-bit, Stable-SPAM achieves the
same loss as Adam while requiring only about half the training steps. Code is
available at https://github.com/TianjinYellow/StableSPAM.git.
Stable-SPAM, an enhanced version of SPAM optimizer, stabilizes gradient norms and improves performance in low-bit precision training across various bit levels.
4-bit training low-bit precision learning rates gradient norms divergence SPAM momentum reset spike-aware gradient clipping Stable-SPAM adaptive clipping threshold historical maxima $l_2$-norm statistics Adam LLM training LLaMA-1B perplexity training steps