D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learning
Offline reinforcement learning algorithms hold the promise of enabling
data-driven RL methods that do not require costly or dangerous real-world
exploration and benefit from large pre-collected datasets. This in turn can
facilitate real-world applications, as well as a more standardized approach to
RL research. Furthermore, offline RL methods can provide effective
initializations for online finetuning to overcome challenges with exploration.
However, evaluating progress on offline RL algorithms requires effective and
challenging benchmarks that capture properties of real-world tasks, provide a
range of task difficulties, and cover a range of challenges both in terms of
the parameters of the domain (e.g., length of the horizon, sparsity of rewards)
and the parameters of the data (e.g., narrow demonstration data or broad
exploratory data). While considerable progress in offline RL in recent years
has been enabled by simpler benchmark tasks, the most widely used datasets are
increasingly saturating in performance and may fail to reflect properties of
realistic tasks. We propose a new benchmark for offline RL that focuses on
realistic simulations of robotic manipulation and locomotion environments,
based on models of real-world robotic systems, and comprising a variety of data
sources, including scripted data, play-style data collected by human
teleoperators, and other data sources. Our proposed benchmark covers
state-based and image-based domains, and supports both offline RL and online
fine-tuning evaluation, with some of the tasks specifically designed to require
both pre-training and fine-tuning. We hope that our proposed benchmark will
facilitate further progress on both offline RL and fine-tuning algorithms.
Website with code, examples, tasks, and data is available at
https://sites.google.com/view/d5rl/
66c2a724a15b4eed7f307fd7
Rafael Rafailov
66c2a724a15b4eed7f307fd8
Kyle Hatch
66c2a724a15b4eed7f307fd9
Anikait Singh
claimed_verified
2024-08-19T22:13:18.609Z
6511ee845b7e52b0251fdee9
https://cdn-avatars.huggingface.co/v1/production/uploads/6511ee845b7e52b0251fdee9/hTIwiIYBGOVnIrxtpri83.png
Anikait Singh
user
Asap7772
66c2a724a15b4eed7f307fda
Laura Smith
66c2a724a15b4eed7f307fdb
Aviral Kumar
66c2a724a15b4eed7f307fdc
Ilya Kostrikov
66c2a724a15b4eed7f307fdd
Philippe Hansen-Estruch
claimed_verified
2024-12-17T08:04:52.912Z
644eb923a8f384abd038a4f7
/avatars/3d1a29c6149a6ffc482842f1438ba55a.svg
Philippe I Hansen-Estruch
user
philippehansen
66c2a724a15b4eed7f307fde
Victor Kolev
66c2a724a15b4eed7f307fdf
Philip Ball
66c2a724a15b4eed7f307fe0
Jiajun Wu
66c2a724a15b4eed7f307fe1
Chelsea Finn
66c2a724a15b4eed7f307fe2
Sergey Levine
D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learning
Offline reinforcement learning algorithms hold the promise of enabling
data-driven RL methods that do not require costly or dangerous real-world
exploration and benefit from large pre-collected datasets. This in turn can
facilitate real-world applications, as well as a more standardized approach to
RL research. Furthermore, offline RL methods can provide effective
initializations for online finetuning to overcome challenges with exploration.
However, evaluating progress on offline RL algorithms requires effective and
challenging benchmarks that capture properties of real-world tasks, provide a
range of task difficulties, and cover a range of challenges both in terms of
the parameters of the domain (e.g., length of the horizon, sparsity of rewards)
and the parameters of the data (e.g., narrow demonstration data or broad
exploratory data). While considerable progress in offline RL in recent years
has been enabled by simpler benchmark tasks, the most widely used datasets are
increasingly saturating in performance and may fail to reflect properties of
realistic tasks. We propose a new benchmark for offline RL that focuses on
realistic simulations of robotic manipulation and locomotion environments,
based on models of real-world robotic systems, and comprising a variety of data
sources, including scripted data, play-style data collected by human
teleoperators, and other data sources. Our proposed benchmark covers
state-based and image-based domains, and supports both offline RL and online
fine-tuning evaluation, with some of the tasks specifically designed to require
both pre-training and fine-tuning. We hope that our proposed benchmark will
facilitate further progress on both offline RL and fine-tuning algorithms.
Website with code, examples, tasks, and data is available at
https://sites.google.com/view/d5rl/
A new benchmark for offline RL emphasizes realistic robotic tasks, supporting both offline training and online fine-tuning with diverse data sources.
offline reinforcement learning real-world simulations robotic manipulation locomotion environments scripted data play-style data human teleoperators state-based domains image-based domains offline RL online fine-tuning