Taming LLMs by Scaling Learning Rates with Gradient Grouping
Training large language models (LLMs) poses challenges due to their massive
scale and heterogeneous architectures. While adaptive optimizers like AdamW
help address gradient variations, they still struggle with efficient and
effective parameter-wise learning rate estimation, resulting in training
instability, slow convergence, and poor compatibility with parameter-efficient
fine-tuning (PEFT) techniques. This work introduces Scaling with Gradient
Grouping (SGG), an optimizer wrapper that improves adaptive learning rate
estimation by dynamic grouping and group-specific scaling. SGG first groups
gradient statistics in each layer into clusters and then applies
cluster-specific scaling to calibrate learning rates for each parameter, thus
imposing collective group-wise constraints while maintaining precise
per-parameter adaptation. Experiments on diverse (M)LLM benchmarks show that
SGG integrates seamlessly with existing optimizers, and offers consistent gains
and faster convergence over baselines, with various model sizes. Its stability
across varying batch sizes and learning rates establishes SGG as a robust
choice for LLM optimization.
683e5b9a1167d9630159b27f
Siyuan Li
claimed_verified
2025-06-03T08:46:39.296Z
640f7083208821a59b74c757
https://cdn-avatars.huggingface.co/v1/production/uploads/1678735253848-640f7083208821a59b74c757.jpeg
Siyuan Li
user
Lupin1998
683e5b9a1167d9630159b280
Juanxi Tian
admin_assigned
2025-06-03T09:40:52.165Z
670880950e79a8b46f7ff9dd
https://cdn-avatars.huggingface.co/v1/production/uploads/670880950e79a8b46f7ff9dd/hA1TLhwlQblkFsq8wLrkB.jpeg
Juanxi Tian
user
Juanxi
683e5b9a1167d9630159b281
Zedong Wang
claimed_verified
2025-06-03T08:46:41.911Z
6594d390674349122ce6f368
https://cdn-avatars.huggingface.co/v1/production/uploads/6594d390674349122ce6f368/KdWz6lZyGYQpjAgBDeiC1.jpeg
Zedong Wang
user
JackyWangAI
683e5b9a1167d9630159b282
Xin Jin
claimed_verified
2025-06-04T09:03:11.830Z
66aa39349238d9c3a1c7f9dc
https://cdn-avatars.huggingface.co/v1/production/uploads/66aa39349238d9c3a1c7f9dc/mj6r7uxEYXM502x296UMf.jpeg
Jin
user
Xin1118
683e5b9a1167d9630159b283
Zicheng Liu
683e5b9a1167d9630159b284
Wentao Zhang
683e5b9a1167d9630159b285
Dan Xu
admin_assigned
2025-06-04T18:57:21.130Z
66feab48651e00e22f33222e
/avatars/7344377e2c796c7ec85194bb2fc78521.svg
Dan Xu
user
danxuhk
Taming LLMs by Scaling Learning Rates with Gradient Grouping
Training large language models (LLMs) poses challenges due to their massive
scale and heterogeneous architectures. While adaptive optimizers like AdamW
help address gradient variations, they still struggle with efficient and
effective parameter-wise learning rate estimation, resulting in training
instability, slow convergence, and poor compatibility with parameter-efficient
fine-tuning (PEFT) techniques. This work introduces Scaling with Gradient
Grouping (SGG), an optimizer wrapper that improves adaptive learning rate
estimation by dynamic grouping and group-specific scaling. SGG first groups
gradient statistics in each layer into clusters and then applies
cluster-specific scaling to calibrate learning rates for each parameter, thus
imposing collective group-wise constraints while maintaining precise
per-parameter adaptation. Experiments on diverse (M)LLM benchmarks show that
SGG integrates seamlessly with existing optimizers, and offers consistent gains
and faster convergence over baselines, with various model sizes. Its stability
across varying batch sizes and learning rates establishes SGG as a robust
choice for LLM optimization.
SGG, an optimizer wrapper, enhances adaptive learning rates for large language models by grouping gradients and applying cluster-specific scaling, improving convergence and stability.
large language models adaptive optimizers AdamW parameter-wise learning rate estimation training instability parameter-efficient fine-tuning Scaling with Gradient Grouping gradient grouping cluster-specific scaling LLM benchmarks robust choice for LLM optimization