DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion
  Models
Learning from human feedback has been shown to improve text-to-image models.
These techniques first learn a reward function that captures what humans care
about in the task and then improve the models based on the learned reward
function. Even though relatively simple approaches (e.g., rejection sampling
based on reward scores) have been investigated, fine-tuning text-to-image
models with the reward function remains challenging. In this work, we propose
using online reinforcement learning (RL) to fine-tune text-to-image models. We
focus on diffusion models, defining the fine-tuning task as an RL problem, and
updating the pre-trained text-to-image diffusion models using policy gradient
to maximize the feedback-trained reward. Our approach, coined DPOK, integrates
policy optimization with KL regularization. We conduct an analysis of KL
regularization for both RL fine-tuning and supervised fine-tuning. In our
experiments, we show that DPOK is generally superior to supervised fine-tuning
with respect to both image-text alignment and image quality.
64741205a7f23affe77887cc
Ying Fan
claimed_verified
2023-05-30T17:59:53.816Z
64761c7595a4dc98e58c5107
/avatars/132c907149c60c60fca77519fbd3d695.svg
Ying Fan
user
yingfanbot
64741205a7f23affe77887cd
Olivia Watkins
64741205a7f23affe77887ce
Yuqing Du
64741205a7f23affe77887cf
Hao Liu
claimed_verified
2023-08-07T12:39:32.469Z
62aa53fc5df2251f7df798c6
/avatars/1c7828ce861c30ddd0afceff355a92ec.svg
Hao
user
haoliu
64741205a7f23affe77887d0
Moonkyung Ryu
64741205a7f23affe77887d1
Craig Boutilier
64741205a7f23affe77887d2
Pieter Abbeel
64741205a7f23affe77887d3
Mohammad Ghavamzadeh
64741205a7f23affe77887d4
Kangwook Lee
claimed_verified
2023-05-30T17:59:56.614Z
6354137306d707b332451421
/avatars/46770f32702e3ad08f91faeef9e4ea6e.svg
Kangwook Lee
user
kw1jjang
64741205a7f23affe77887d5
Kimin Lee
claimed_verified
2023-05-30T17:59:50.849Z
635097ec59bfa9a85d4207b2
/avatars/787085894e9e6538b6b3e3051efe9eea.svg
Kimin Lee
user
kiminle2
DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion
  Models
Learning from human feedback has been shown to improve text-to-image models.
These techniques first learn a reward function that captures what humans care
about in the task and then improve the models based on the learned reward
function. Even though relatively simple approaches (e.g., rejection sampling
based on reward scores) have been investigated, fine-tuning text-to-image
models with the reward function remains challenging. In this work, we propose
using online reinforcement learning (RL) to fine-tune text-to-image models. We
focus on diffusion models, defining the fine-tuning task as an RL problem, and
updating the pre-trained text-to-image diffusion models using policy gradient
to maximize the feedback-trained reward. Our approach, coined DPOK, integrates
policy optimization with KL regularization. We conduct an analysis of KL
regularization for both RL fine-tuning and supervised fine-tuning. In our
experiments, we show that DPOK is generally superior to supervised fine-tuning
with respect to both image-text alignment and image quality.
Using online reinforcement learning with policy gradient and KL regularization to fine-tune text-to-image diffusion models improves image-text alignment and image quality compared to supervised fine-tuning.
reward function text-to-image models online reinforcement learning RL diffusion models fine-tuning policy gradient KL regularization DPOK image-text alignment image quality supervised fine-tuning