Training Diffusion Models with Reinforcement Learning
Diffusion models are a class of flexible generative models trained with an
approximation to the log-likelihood objective. However, most use cases of
diffusion models are not concerned with likelihoods, but instead with
downstream objectives such as human-perceived image quality or drug
effectiveness. In this paper, we investigate reinforcement learning methods for
directly optimizing diffusion models for such objectives. We describe how
posing denoising as a multi-step decision-making problem enables a class of
policy gradient algorithms, which we refer to as denoising diffusion policy
optimization (DDPO), that are more effective than alternative reward-weighted
likelihood approaches. Empirically, DDPO is able to adapt text-to-image
diffusion models to objectives that are difficult to express via prompting,
such as image compressibility, and those derived from human feedback, such as
aesthetic quality. Finally, we show that DDPO can improve prompt-image
alignment using feedback from a vision-language model without the need for
additional data collection or human annotation.
646c1548e5abcbf670a39f0e
Kevin Black
claimed_verified
2023-06-28T20:14:43.276Z
646fde2c799a974be31b6031
/avatars/d816aa2d18930f4a634096f1c82a49bf.svg
Kevin Black
user
kvablack
646c1548e5abcbf670a39f0f
Michael Janner
admin_assigned
2023-05-24T08:32:10.698Z
6317f8c7aea37e1333e3d476
/avatars/a2eb33dafc3b475e02a1ad182a964c1b.svg
Michael Janner
user
janner
646c1548e5abcbf670a39f10
Yilun Du
admin_assigned
2023-05-24T08:31:34.506Z
63c9bd445fdc575773c732fe
/avatars/def472d1ab3fbf751225357c0932ae7e.svg
Yilun Du
user
yilundu
646c1548e5abcbf670a39f11
Ilya Kostrikov
646c1548e5abcbf670a39f12
Sergey Levine
Training Diffusion Models with Reinforcement Learning
Diffusion models are a class of flexible generative models trained with an
approximation to the log-likelihood objective. However, most use cases of
diffusion models are not concerned with likelihoods, but instead with
downstream objectives such as human-perceived image quality or drug
effectiveness. In this paper, we investigate reinforcement learning methods for
directly optimizing diffusion models for such objectives. We describe how
posing denoising as a multi-step decision-making problem enables a class of
policy gradient algorithms, which we refer to as denoising diffusion policy
optimization (DDPO), that are more effective than alternative reward-weighted
likelihood approaches. Empirically, DDPO is able to adapt text-to-image
diffusion models to objectives that are difficult to express via prompting,
such as image compressibility, and those derived from human feedback, such as
aesthetic quality. Finally, we show that DDPO can improve prompt-image
alignment using feedback from a vision-language model without the need for
additional data collection or human annotation.
Reinforcement learning method DDPO optimizes diffusion models directly for downstream objectives such as image quality and feedback-driven alignment.
diffusion models log-likelihood objective denoising policy gradient algorithms denoising diffusion policy optimization (DDPO) text-to-image diffusion models image compressibility aesthetic quality prompt-image alignment vision-language model