Benchmarking Optimizers for Large Language Model Pretraining
The recent development of Large Language Models (LLMs) has been accompanied
by an effervescence of novel ideas and methods to better optimize the loss of
deep learning models. Claims from those methods are myriad: from faster
convergence to removing reliance on certain hyperparameters. However, the
diverse experimental protocols used to validate these claims make direct
comparisons between methods challenging. This study presents a comprehensive
evaluation of recent optimization techniques across standardized LLM
pretraining scenarios, systematically varying model size, batch size, and
training duration. Through careful tuning of each method, we provide guidance
to practitioners on which optimizer is best suited for each scenario. For
researchers, our work highlights promising directions for future optimization
research. Finally, by releasing our code and making all experiments fully
reproducible, we hope our efforts can help the development and rigorous
benchmarking of future methods.
68b7df47295f15ff609114b5
Andrei Semenov
claimed_verified
2025-09-03T08:26:03.296Z
6479f8335f3450e1ded40774
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/iOjE2dcSUS-XYE0WJe6U8.jpeg
Andrei Semenov
user
Andron00e
68b7df47295f15ff609114b6
Matteo Pagliardini
admin_assigned
2025-09-03T08:50:10.068Z
647e2802e4d52fe0e01f1f33
/avatars/38a6c66bce327eded16993932b62986b.svg
Matteo Pagliardini
user
MatPag
68b7df47295f15ff609114b7
Martin Jaggi
admin_assigned
2025-09-03T08:50:15.955Z
654baf61d625e083383dfd00
/avatars/e933b2b9d7226e9bb72f77916c2be4f1.svg
Martin Jaggi
user
mjaggi
Benchmarking Optimizers for Large Language Model Pretraining
The recent development of Large Language Models (LLMs) has been accompanied
by an effervescence of novel ideas and methods to better optimize the loss of
deep learning models. Claims from those methods are myriad: from faster
convergence to removing reliance on certain hyperparameters. However, the
diverse experimental protocols used to validate these claims make direct
comparisons between methods challenging. This study presents a comprehensive
evaluation of recent optimization techniques across standardized LLM
pretraining scenarios, systematically varying model size, batch size, and
training duration. Through careful tuning of each method, we provide guidance
to practitioners on which optimizer is best suited for each scenario. For
researchers, our work highlights promising directions for future optimization
research. Finally, by releasing our code and making all experiments fully
reproducible, we hope our efforts can help the development and rigorous
benchmarking of future methods.
A comprehensive evaluation of recent optimization techniques for Large Language Models provides guidance on selecting the best optimizer for different pretraining scenarios.
Large Language Models LLMs optimization techniques pretraining scenarios model size batch size training duration optimizer