The Surprising Agreement Between Convex Optimization Theory and
  Learning-Rate Scheduling for Large Model Training
We show that learning-rate schedules for large model training behave
surprisingly similar to a performance bound from non-smooth convex optimization
theory. We provide a bound for the constant schedule with linear cooldown; in
particular, the practical benefit of cooldown is reflected in the bound due to
the absence of logarithmic terms. Further, we show that this surprisingly close
match between optimization theory and practice can be exploited for
learning-rate tuning: we achieve noticeable improvements for training 124M and
210M Llama-type models by (i) extending the schedule for continued training
with optimal learning-rate, and (ii) transferring the optimal learning-rate
across schedules.
67a0e794042d0e5936db83cf
Fabian Schaipp
claimed_verified
2025-02-03T16:00:56.014Z
64a2b68da0696e0a29739349
https://cdn-avatars.huggingface.co/v1/production/uploads/64a2b68da0696e0a29739349/wUtx3yCcSiN7SZP4nMSPK.png
Fabian S
user
fabian-sp
67a0e794042d0e5936db83d0
Alexander Hägele
admin_assigned
2025-02-03T16:01:03.085Z
65785d22dddc2360b01702e1
/avatars/8e3ddf25b9c423f57484fddef4f0aafd.svg
Alexander Hägele
user
haeggee
67a0e794042d0e5936db83d1
Adrien Taylor
67a0e794042d0e5936db83d2
Umut Simsekli
67a0e794042d0e5936db83d3
Francis Bach
The Surprising Agreement Between Convex Optimization Theory and
  Learning-Rate Scheduling for Large Model Training
We show that learning-rate schedules for large model training behave
surprisingly similar to a performance bound from non-smooth convex optimization
theory. We provide a bound for the constant schedule with linear cooldown; in
particular, the practical benefit of cooldown is reflected in the bound due to
the absence of logarithmic terms. Further, we show that this surprisingly close
match between optimization theory and practice can be exploited for
learning-rate tuning: we achieve noticeable improvements for training 124M and
210M Llama-type models by (i) extending the schedule for continued training
with optimal learning-rate, and (ii) transferring the optimal learning-rate
across schedules.
Learning-rate schedules for large deep learning models align closely with non-smooth convex optimization theory, allowing for improved training and tuning.
learning-rate schedules non-smooth convex optimization linear cooldown optimal learning-rate Llama-type models