Diffusion Policy Policy Optimization
We introduce Diffusion Policy Policy Optimization, DPPO, an algorithmic
framework including best practices for fine-tuning diffusion-based policies
(e.g. Diffusion Policy) in continuous control and robot learning tasks using
the policy gradient (PG) method from reinforcement learning (RL). PG methods
are ubiquitous in training RL policies with other policy parameterizations;
nevertheless, they had been conjectured to be less efficient for
diffusion-based policies. Surprisingly, we show that DPPO achieves the
strongest overall performance and efficiency for fine-tuning in common
benchmarks compared to other RL methods for diffusion-based policies and also
compared to PG fine-tuning of other policy parameterizations. Through
experimental investigation, we find that DPPO takes advantage of unique
synergies between RL fine-tuning and the diffusion parameterization, leading to
structured and on-manifold exploration, stable training, and strong policy
robustness. We further demonstrate the strengths of DPPO in a range of
realistic settings, including simulated robotic tasks with pixel observations,
and via zero-shot deployment of simulation-trained policies on robot hardware
in a long-horizon, multi-stage manipulation task. Website with code:
diffusion-ppo.github.io
66d80194e1aee6b7147925d9
Allen Z. Ren
claimed_verified
2024-09-04T16:21:05.487Z
6392628841491be61b52a4dc
/avatars/c1e59fbf2252e332a3c86c2bd18afd09.svg
Allen Z. Ren
user
allenzren
66d80194e1aee6b7147925da
Justin Lidard
admin_assigned
2024-09-04T09:21:49.498Z
6570fb5b637aa903549d12d7
/avatars/992fe518881362bfa71d04987fcccdd0.svg
Justin Lidard
user
jlidard
66d80194e1aee6b7147925db
Lars L. Ankile
claimed_verified
2024-09-05T15:31:19.092Z
637d214ab61b6d662aeefe09
https://cdn-avatars.huggingface.co/v1/production/uploads/637d214ab61b6d662aeefe09/DADs1QHxi-Sa7P6Ba25Kw.jpeg
Lars Lien Ankile
user
ankile
66d80194e1aee6b7147925dc
Anthony Simeonov
66d80194e1aee6b7147925dd
Pulkit Agrawal
admin_assigned
2024-09-04T09:26:27.275Z
65314176e5b85656bd4a5d72
/avatars/c62534c0920d90ecb5d4b1506d242e9f.svg
Pulkit Agrawal
user
pulkitag
66d80194e1aee6b7147925de
Anirudha Majumdar
66d80194e1aee6b7147925df
Benjamin Burchfiel
66d80194e1aee6b7147925e0
Hongkai Dai
admin_assigned
2024-09-04T09:25:37.305Z
6614c0e00bbea65e71d7efbe
/avatars/126f3548a15604d28722ff99b6608f8f.svg
Hongkai Dai
user
daihongkai
66d80194e1aee6b7147925e1
Max Simchowitz
claimed_verified
2024-09-05T07:24:35.152Z
66d89d802c646e71e7eab6bf
/avatars/008b57235d6056751550dc9a93576e9d.svg
Max Simchowitz
user
msimchowitz
Diffusion Policy Policy Optimization
We introduce Diffusion Policy Policy Optimization, DPPO, an algorithmic
framework including best practices for fine-tuning diffusion-based policies
(e.g. Diffusion Policy) in continuous control and robot learning tasks using
the policy gradient (PG) method from reinforcement learning (RL). PG methods
are ubiquitous in training RL policies with other policy parameterizations;
nevertheless, they had been conjectured to be less efficient for
diffusion-based policies. Surprisingly, we show that DPPO achieves the
strongest overall performance and efficiency for fine-tuning in common
benchmarks compared to other RL methods for diffusion-based policies and also
compared to PG fine-tuning of other policy parameterizations. Through
experimental investigation, we find that DPPO takes advantage of unique
synergies between RL fine-tuning and the diffusion parameterization, leading to
structured and on-manifold exploration, stable training, and strong policy
robustness. We further demonstrate the strengths of DPPO in a range of
realistic settings, including simulated robotic tasks with pixel observations,
and via zero-shot deployment of simulation-trained policies on robot hardware
in a long-horizon, multi-stage manipulation task. Website with code:
diffusion-ppo.github.io
DPPO, a policy gradient method for fine-tuning diffusion-based policies, outperforms other RL methods in continuous control and robot learning tasks, offering stable training, robust policies, and effective zero-shot deployment.
Diffusion Policy Policy Optimization DPPO diffusion-based policies policy gradient reinforcement learning fine-tuning structured exploration on-manifold exploration stable training policy robustness zero-shot deployment simulation-trained policies