Value-Based Deep RL Scales Predictably
Scaling data and compute is critical to the success of machine learning.
However, scaling demands predictability: we want methods to not only perform
well with more compute or data, but also have their performance be predictable
from small-scale runs, without running the large-scale experiment. In this
paper, we show that value-based off-policy RL methods are predictable despite
community lore regarding their pathological behavior. First, we show that data
and compute requirements to attain a given performance level lie on a Pareto
frontier, controlled by the updates-to-data (UTD) ratio. By estimating this
frontier, we can predict this data requirement when given more compute, and
this compute requirement when given more data. Second, we determine the optimal
allocation of a total resource budget across data and compute for a given
performance and use it to determine hyperparameters that maximize performance
for a given budget. Third, this scaling behavior is enabled by first estimating
predictable relationships between hyperparameters, which is used to manage
effects of overfitting and plasticity loss unique to RL. We validate our
approach using three algorithms: SAC, BRO, and PQL on DeepMind Control, OpenAI
gym, and IsaacGym, when extrapolating to higher levels of data, compute,
budget, or performance.
67aa44a0927861da2b7a3479
Oleh Rybkin
extracted_pending
2025-02-10T18:25:37.833Z
64d1161315b26cc7f70f37e6
/avatars/c020b7d2c6c2bb1ca289a9cf0c4eaf00.svg
Oleh Rybkin
user
orybkin
67aa44a0927861da2b7a347a
Michal Nauman
67aa44a0927861da2b7a347b
Preston Fu
67aa44a0927861da2b7a347c
Charlie Snell
67aa44a0927861da2b7a347d
Pieter Abbeel
67aa44a0927861da2b7a347e
Sergey Levine
67aa44a0927861da2b7a347f
Aviral Kumar
claimed_verified
2025-02-11T14:25:38.016Z
67315a324d2eab8035de786a
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/INH5_PlcJYfYcGkyibjgP.png
Aviral Kumar
user
aviralku
Value-Based Deep RL Scales Predictably
Scaling data and compute is critical to the success of machine learning.
However, scaling demands predictability: we want methods to not only perform
well with more compute or data, but also have their performance be predictable
from small-scale runs, without running the large-scale experiment. In this
paper, we show that value-based off-policy RL methods are predictable despite
community lore regarding their pathological behavior. First, we show that data
and compute requirements to attain a given performance level lie on a Pareto
frontier, controlled by the updates-to-data (UTD) ratio. By estimating this
frontier, we can predict this data requirement when given more compute, and
this compute requirement when given more data. Second, we determine the optimal
allocation of a total resource budget across data and compute for a given
performance and use it to determine hyperparameters that maximize performance
for a given budget. Third, this scaling behavior is enabled by first estimating
predictable relationships between hyperparameters, which is used to manage
effects of overfitting and plasticity loss unique to RL. We validate our
approach using three algorithms: SAC, BRO, and PQL on DeepMind Control, OpenAI
gym, and IsaacGym, when extrapolating to higher levels of data, compute,
budget, or performance.
Value-based off-policy RL methods exhibit predictable scaling behavior based on the updates-to-data ratio, enabling optimal resource allocation and performance maximization for specific budgets.
value-based off-policy RL Pareto frontier updates-to-data ratio DeepMind Control OpenAI gym IsaacGym