Fantastic Pretraining Optimizers and Where to Find Them
AdamW has long been the dominant optimizer in language model pretraining,
despite numerous claims that alternative optimizers offer 1.4 to 2x speedup. We
posit that two methodological shortcomings have obscured fair comparisons and
hindered practical adoption: (i) unequal hyperparameter tuning and (ii) limited
or misleading evaluation setups. To address these two issues, we conduct a
systematic study of ten deep learning optimizers across four model scales
(0.1B-1.2B parameters) and data-to-model ratios (1-8x the Chinchilla optimum).
We find that fair and informative comparisons require rigorous hyperparameter
tuning and evaluations across a range of model scales and data-to-model ratios,
performed at the end of training. First, optimal hyperparameters for one
optimizer may be suboptimal for another, making blind hyperparameter transfer
unfair. Second, the actual speedup of many proposed optimizers over well-tuned
baselines is lower than claimed and decreases with model size to only 1.1x for
1.2B parameter models. Thirdly, comparing intermediate checkpoints before
reaching the target training budgets can be misleading, as rankings between two
optimizers can flip during training due to learning rate decay. Through our
thorough investigation, we find that all the fastest optimizers such as Muon
and Soap, use matrices as preconditioners -- multiplying gradients with
matrices rather than entry-wise scalars. However, the speedup of matrix-based
optimizers is inversely proportional to model scale, decreasing from 1.4x over
AdamW for 0.1B parameter models to merely 1.1x for 1.2B parameter models.
68b7efa6295f15ff6091151e
Kaiyue Wen
68b7efa6295f15ff6091151f
David Hall
68b7efa6295f15ff60911520
Tengyu Ma
68b7efa6295f15ff60911521
Percy Liang
Fantastic Pretraining Optimizers and Where to Find Them
AdamW has long been the dominant optimizer in language model pretraining,
despite numerous claims that alternative optimizers offer 1.4 to 2x speedup. We
posit that two methodological shortcomings have obscured fair comparisons and
hindered practical adoption: (i) unequal hyperparameter tuning and (ii) limited
or misleading evaluation setups. To address these two issues, we conduct a
systematic study of ten deep learning optimizers across four model scales
(0.1B-1.2B parameters) and data-to-model ratios (1-8x the Chinchilla optimum).
We find that fair and informative comparisons require rigorous hyperparameter
tuning and evaluations across a range of model scales and data-to-model ratios,
performed at the end of training. First, optimal hyperparameters for one
optimizer may be suboptimal for another, making blind hyperparameter transfer
unfair. Second, the actual speedup of many proposed optimizers over well-tuned
baselines is lower than claimed and decreases with model size to only 1.1x for
1.2B parameter models. Thirdly, comparing intermediate checkpoints before
reaching the target training budgets can be misleading, as rankings between two
optimizers can flip during training due to learning rate decay. Through our
thorough investigation, we find that all the fastest optimizers such as Muon
and Soap, use matrices as preconditioners -- multiplying gradients with
matrices rather than entry-wise scalars. However, the speedup of matrix-based
optimizers is inversely proportional to model scale, decreasing from 1.4x over
AdamW for 0.1B parameter models to merely 1.1x for 1.2B parameter models.
A systematic study reveals that fair comparisons of deep learning optimizers require rigorous hyperparameter tuning and evaluations across various model scales and data-to-model ratios, showing that matrix-based optimizers like Muon and Soap offer diminishing speedups as model size increases.
AdamW deep learning optimizers hyperparameter tuning evaluation setups model scales data-to-model ratios Muon Soap preconditioners learning rate decay