Tune As You Scale: Hyperparameter Optimization For Compute Efficient
  Training
Hyperparameter tuning of deep learning models can lead to order-of-magnitude
performance gains for the same amount of compute. Despite this, systematic
tuning is uncommon, particularly for large models, which are expensive to
evaluate and tend to have many hyperparameters, necessitating difficult
judgment calls about tradeoffs, budgets, and search bounds. To address these
issues and propose a practical method for robustly tuning large models, we
present Cost-Aware Pareto Region Bayesian Search (CARBS), a Bayesian
optimization algorithm that performs local search around the performance-cost
Pareto frontier. CARBS does well even in unbounded search spaces with many
hyperparameters, learns scaling relationships so that it can tune models even
as they are scaled up, and automates much of the "black magic" of tuning. Among
our results, we effectively solve the entire ProcGen benchmark just by tuning a
simple baseline (PPO, as provided in the original ProcGen paper). We also
reproduce the model size vs. training tokens scaling result from the Chinchilla
project (Hoffmann et al. 2022), while simultaneously discovering scaling laws
for every other hyperparameter, via an easy automated process that uses
significantly less compute and is applicable to any deep learning problem (not
just language models).
648bc57995d85bfac02180d9
Abraham J. Fetterman
648bc57995d85bfac02180da
Ellie Kitanidis
648bc57995d85bfac02180db
Joshua Albrecht
648bc57995d85bfac02180dc
Zachary Polizzi
648bc57995d85bfac02180dd
Bryden Fogelman
648bc57995d85bfac02180de
Maksis Knutins
648bc57995d85bfac02180df
Bartosz Wróblewski
648bc57995d85bfac02180e0
James B. Simon
648bc57995d85bfac02180e1
Kanjun Qiu
Tune As You Scale: Hyperparameter Optimization For Compute Efficient
  Training
Hyperparameter tuning of deep learning models can lead to order-of-magnitude
performance gains for the same amount of compute. Despite this, systematic
tuning is uncommon, particularly for large models, which are expensive to
evaluate and tend to have many hyperparameters, necessitating difficult
judgment calls about tradeoffs, budgets, and search bounds. To address these
issues and propose a practical method for robustly tuning large models, we
present Cost-Aware Pareto Region Bayesian Search (CARBS), a Bayesian
optimization algorithm that performs local search around the performance-cost
Pareto frontier. CARBS does well even in unbounded search spaces with many
hyperparameters, learns scaling relationships so that it can tune models even
as they are scaled up, and automates much of the "black magic" of tuning. Among
our results, we effectively solve the entire ProcGen benchmark just by tuning a
simple baseline (PPO, as provided in the original ProcGen paper). We also
reproduce the model size vs. training tokens scaling result from the Chinchilla
project (Hoffmann et al. 2022), while simultaneously discovering scaling laws
for every other hyperparameter, via an easy automated process that uses
significantly less compute and is applicable to any deep learning problem (not
just language models).
A Bayesian optimization algorithm, Cost-Aware Pareto Region Bayesian Search (CARBS), enables robust tuning of large deep learning models, solving ProcGen and discovering scaling laws with less compute.
Bayesian optimization CARBS performance-cost Pareto frontier scaling relationships ProcGen benchmark Chinchilla project model size training tokens hyperparameter tuning