Compute-Optimal Scaling for Value-Based Deep RL
Preston Fu, Oleh Rybkin, Zhiyuan Zhou, Michal Nauman, Pieter Abbeel, Sergey Levine, Aviral Kumar
As models become bigger and more powerful, it becomes increasingly essential to scale up training recipes to not just be bigger along every dimension, but to be compute optimal, in the sense that maximal benefit is extracted per unit of compute as a function of model size, gradient steps, batch size, and other hyperparameters. While compute optimal scaling has been explored extensively for supervised and unsupervised pre-training, it has received comparatively less attention for reinforcement learning (RL). In this paper, we study compute scaling for online, value-based deep RL. These methods present two axes for compute allocation: the capacity of the model and update-to-data (UTD) ratio. Given a total compute budget, we investigate how to best partition resources between these two axes to maximize performance or sample efficiency. In the process, we observe an interesting interplay between model size, UTD, and batch size, leading us to a phenomenon we call ``TD-overfitting'', where large batch size training results in worse Q-functions when smaller networks are used. Interestingly, we find that the presence of this sort of TD-overfitting reverses with larger models, enabling the use of greater batch sizes for training.  We develop a mechanistic understanding of these interactions, allowing us to identify effective batch sizes when operating at scale. Our results lead to empirically validated prescriptions for compute-optimal scaling in deep RL, analogous to scaling laws for compute allocation in supervised learning but tailored to the unique dynamics of TD-learning.