Cautious Weight Decay
We introduce Cautious Weight Decay (CWD), a one-line, optimizer-agnostic
modification that applies weight decay only to parameter coordinates whose
signs align with the optimizer update. Unlike standard decoupled decay, which
implicitly optimizes a regularized or constrained objective, CWD preserves the
original loss and admits a bilevel interpretation: it induces sliding-mode
behavior upon reaching the stationary manifold, allowing it to search for
locally Pareto-optimal stationary points of the unmodified objective. In
practice, CWD is a drop-in change for optimizers such as AdamW, Lion, and Muon,
requiring no new hyperparameters or additional tuning. For language model
pre-training and ImageNet classification, CWD consistently improves final loss
and accuracy at million- to billion-parameter scales.
68efc200d22df134b7f5ca31
Lizhang Chen
claimed_verified
2025-10-17T04:14:14.152Z
657ac67b5b9c86c08f2c073d
/avatars/a6770776bba02413123d53eb020240b4.svg
'Tim' Lizhang Chen
user
lzchenut
68efc200d22df134b7f5ca32
Jonathan Li
68efc200d22df134b7f5ca33
Kaizhao Liang
68efc200d22df134b7f5ca34
Baiyu Su
68efc200d22df134b7f5ca35
Cong Xie
68efc200d22df134b7f5ca36
Nuo Wang Pierse
68efc200d22df134b7f5ca37
Chen Liang
68efc200d22df134b7f5ca38
Ni Lao
68efc200d22df134b7f5ca39
Qiang Liu
Cautious Weight Decay
We introduce Cautious Weight Decay (CWD), a one-line, optimizer-agnostic
modification that applies weight decay only to parameter coordinates whose
signs align with the optimizer update. Unlike standard decoupled decay, which
implicitly optimizes a regularized or constrained objective, CWD preserves the
original loss and admits a bilevel interpretation: it induces sliding-mode
behavior upon reaching the stationary manifold, allowing it to search for
locally Pareto-optimal stationary points of the unmodified objective. In
practice, CWD is a drop-in change for optimizers such as AdamW, Lion, and Muon,
requiring no new hyperparameters or additional tuning. For language model
pre-training and ImageNet classification, CWD consistently improves final loss
and accuracy at million- to billion-parameter scales.
Cautious Weight Decay (CWD) enhances optimizer performance by applying weight decay selectively, improving accuracy and loss in large-scale models without additional tuning.
Cautious Weight Decay CWD optimizer-agnostic weight decay parameter coordinates optimizer update standard decoupled decay bilevel interpretation sliding-mode behavior stationary manifold locally Pareto-optimal AdamW Lion Muon language model pre-training ImageNet classification