Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models
Wancong Zhang, Uladzislau Sobal, Kyunghyun Cho, Randall Balestriero, Tim G. J. Rudner, Yann LeCun
A long-standing goal in AI is to develop agents capable of solving diverse tasks across a range of environments, including those never seen during training. Two dominant paradigms address this challenge: (i) reinforcement learning (RL), which learns policies via trial and error, and (ii) optimal control, which plans actions using a known or learned dynamics model. However, their comparative strengths in the offline setting—where agents must learn from reward-free trajectories—remain underexplored. In this work, we systematically evaluate RL and control-based methods on a suite of navigation tasks, using offline datasets of varying quality. On the RL side, we consider goal-conditioned and zero-shot approaches. On the control side, we train a latent dynamics model using the Joint Embedding Predictive Architecture (JEPA) and employ it for planning. We investigate how factors such as data diversity, trajectory quality, and environment variability influence the performance of these approaches. Our findings reveal that model-free RL excels when abundant, high-quality data is available, while model-based planning demonstrates superior generalization to novel layouts, better trajectory stitching, and greater data efficiency. Notably, planning with a latent dynamics model proves to be a strong approach for handling suboptimal offline data and adapting to diverse environment configurations.