Horizon Reduction Makes Offline RL Scalable
Seohong Park, Kevin Frans, Deepinder Mann, Benjamin Eysenbach, Aviral Kumar, Sergey Levine
In this work, we study the scalability of offline reinforcement learning (RL) algorithms. In principle, a truly scalable offline RL algorithm should be able to solve any given problem, regardless of its complexity, given sufficient data, compute, and model capacity. We investigate if and how current offline RL algorithms match up to this promise on diverse, challenging, previously unsolved tasks, using datasets up to 1000× larger than typical offline RL datasets. We observe that despite scaling up data, many existing offline RL algorithms exhibit poor scaling behavior, saturating well below the maximum performance. We hypothesize that the horizon is the main cause behind the poor scaling of offline RL. Through several analysis experiments, we show that even when controlling for other factors, long horizons can present a fundamental barrier to scaling up offline RL. We then show that various horizon reduction techniques can substantially enhance scalability on challenging tasks. Finally, we propose SHARSA, a minimalistic yet scalable method that effectively reduces the horizon, and show that it achieves the best asymptotic performance and scaling behavior among our evaluation methods.