Offline Goal Conditioned Reinforcement Learning with Temporal Distance Representations
Vivek Myers, Bill Zheng, Benjamin Eysenbach, Sergey Levine
Learned successor features provide a powerful framework for learning goal-reaching policies. These representations are constructed such that similarity in the representation space predicts future outcomes, allowing goal-reaching policies to be extracted. Representations learned for forward inference have some practical limitations - stitching of behaviors does not arise naturally with forward objectives like contrastive classification, and additional regularization is required to enable valid policy extraction. In this work, we propose a new representation learning objective that enables extraction of goal-reaching policies. Our key insight is that rather than learning representations of the future, we should really learn representations that can associate outcomes with preceding states. We show that when combined with existing quasimetric network parameterization and the right invariances, these representations let us learn optimal goal-reaching policies from offline data. On existing offline GCRL benchmarks, the hindsight classification objective improves performance with a simpler algorithm and fewer independent networks/parameters to learn relative to past methods.