Leveraging Skills from Unlabeled Prior Data for Efficient Online
  Exploration
Unsupervised pretraining has been transformative in many supervised domains.
However, applying such ideas to reinforcement learning (RL) presents a unique
challenge in that fine-tuning does not involve mimicking task-specific data,
but rather exploring and locating the solution through iterative
self-improvement. In this work, we study how unlabeled prior trajectory data
can be leveraged to learn efficient exploration strategies. While prior data
can be used to pretrain a set of low-level skills, or as additional off-policy
data for online RL, it has been unclear how to combine these ideas effectively
for online exploration. Our method SUPE (Skills from Unlabeled Prior data for
Exploration) demonstrates that a careful combination of these ideas compounds
their benefits. Our method first extracts low-level skills using a variational
autoencoder (VAE), and then pseudo-relabels unlabeled trajectories using an
optimistic reward model, transforming prior data into high-level, task-relevant
examples. Finally, SUPE uses these transformed examples as additional
off-policy data for online RL to learn a high-level policy that composes
pretrained low-level skills to explore efficiently. We empirically show that
SUPE reliably outperforms prior strategies, successfully solving a suite of
long-horizon, sparse-reward tasks. Code: https://github.com/rail-berkeley/supe.
6719e6dccc4568fb62d392ec
Max Wilcoxson
admin_assigned
2024-10-28T15:49:01.754Z
6568d51a16053f9a304585f9
/avatars/2dc0596e321c978b9513cf5f67141c2a.svg
Max Wilcoxson
user
mwilcoxson
6719e6dccc4568fb62d392ed
Qiyang Li
6719e6dccc4568fb62d392ee
Kevin Frans
admin_assigned
2024-10-28T15:48:46.738Z
6321e71ca97afe3c4c66f7b5
/avatars/02cf0d5779417b37e34cc6340bffacad.svg
Kevin Frans
user
kvfrans
6719e6dccc4568fb62d392ef
Sergey Levine
admin_assigned
2024-10-28T15:48:55.303Z
665ce54120a307a3754849dd
/avatars/e698726e9be61dd50ce2efe372ed5dac.svg
Sergey Levine
user
svlevine
Leveraging Skills from Unlabeled Prior Data for Efficient Online
  Exploration
Unsupervised pretraining has been transformative in many supervised domains.
However, applying such ideas to reinforcement learning (RL) presents a unique
challenge in that fine-tuning does not involve mimicking task-specific data,
but rather exploring and locating the solution through iterative
self-improvement. In this work, we study how unlabeled prior trajectory data
can be leveraged to learn efficient exploration strategies. While prior data
can be used to pretrain a set of low-level skills, or as additional off-policy
data for online RL, it has been unclear how to combine these ideas effectively
for online exploration. Our method SUPE (Skills from Unlabeled Prior data for
Exploration) demonstrates that a careful combination of these ideas compounds
their benefits. Our method first extracts low-level skills using a variational
autoencoder (VAE), and then pseudo-relabels unlabeled trajectories using an
optimistic reward model, transforming prior data into high-level, task-relevant
examples. Finally, SUPE uses these transformed examples as additional
off-policy data for online RL to learn a high-level policy that composes
pretrained low-level skills to explore efficiently. We empirically show that
SUPE reliably outperforms prior strategies, successfully solving a suite of
long-horizon, sparse-reward tasks. Code: https://github.com/rail-berkeley/supe.
SUPE uses unlabeled trajectory data and a variational autoencoder to learn low-level skills and high-level exploration strategies in reinforcement learning.
variational autoencoder optimistic reward model offline RL online RL long-horizon tasks sparse-reward tasks