Diffusion Model Alignment Using Direct Preference Optimization
Large language models (LLMs) are fine-tuned using human comparison data with
Reinforcement Learning from Human Feedback (RLHF) methods to make them better
aligned with users' preferences. In contrast to LLMs, human preference learning
has not been widely explored in text-to-image diffusion models; the best
existing approach is to fine-tune a pretrained model using carefully curated
high quality images and captions to improve visual appeal and text alignment.
We propose Diffusion-DPO, a method to align diffusion models to human
preferences by directly optimizing on human comparison data. Diffusion-DPO is
adapted from the recently developed Direct Preference Optimization (DPO), a
simpler alternative to RLHF which directly optimizes a policy that best
satisfies human preferences under a classification objective. We re-formulate
DPO to account for a diffusion model notion of likelihood, utilizing the
evidence lower bound to derive a differentiable objective. Using the Pick-a-Pic
dataset of 851K crowdsourced pairwise preferences, we fine-tune the base model
of the state-of-the-art Stable Diffusion XL (SDXL)-1.0 model with
Diffusion-DPO. Our fine-tuned base model significantly outperforms both base
SDXL-1.0 and the larger SDXL-1.0 model consisting of an additional refinement
model in human evaluation, improving visual appeal and prompt alignment. We
also develop a variant that uses AI feedback and has comparable performance to
training on human preferences, opening the door for scaling of diffusion model
alignment methods.
655ee5917b44c113b8367213
Bram Wallace
admin_assigned
2023-11-23T10:39:25.309Z
62965a7b7d586f66c392221b
/avatars/c5f17cc83646aa2c2101c91f4eca138d.svg
Bram Wallace
user
bramw
655ee5917b44c113b8367214
Meihua Dang
admin_assigned
2023-11-23T10:39:34.028Z
652eccd3aeb9826ab9cef207
/avatars/2172ec9a2e5f94f006c9a4b8afa01112.svg
Meihua Dang
user
mhdang
655ee5917b44c113b8367215
Rafael Rafailov
admin_assigned
2023-11-23T10:39:41.735Z
64701c98d742e9ef6522598f
/avatars/df3a52bd4bc374d7cb4061f449f75c07.svg
Rafael Rafailov
user
rmrafailov
655ee5917b44c113b8367216
Linqi Zhou
admin_assigned
2023-11-23T10:39:49.966Z
64554c3c9d37c3fb332975ed
/avatars/13001d48b56cca99471a7e75116b70cf.svg
Linqi Zhou
user
alexzhou907
655ee5917b44c113b8367217
Aaron Lou
655ee5917b44c113b8367218
Senthil Purushwalkam
admin_assigned
2023-11-23T10:40:15.660Z
637e633366a51ca5d960a9b1
/avatars/e53242e4e36875105be087c46a552e29.svg
Senthil Purushwalkam
user
senthilps8
655ee5917b44c113b8367219
Stefano Ermon
admin_assigned
2023-11-23T10:40:22.750Z
62f6e244329d4d014d1f4ac5
/avatars/5a8b2bb063c2ebc340504b22530f6811.svg
Stefano Ermon
user
ermonste
655ee5917b44c113b836721a
Caiming Xiong
admin_assigned
2023-11-23T10:40:29.708Z
649dbcc4e0fff1ed099dc80a
/avatars/c87c273ca628dbcddccbf1ee19b2ce33.svg
Caiming Xiong
user
cxiong
655ee5917b44c113b836721b
Shafiq Joty
655ee5917b44c113b836721c
Nikhil Naik
admin_assigned
2023-11-23T10:40:49.932Z
627c2c751b02d7b5bf663fb0
/avatars/f33ac735e2cc910d2da1286d476de5db.svg
Nikhil Naik
user
nikdnaik
Diffusion Model Alignment Using Direct Preference Optimization
Large language models (LLMs) are fine-tuned using human comparison data with
Reinforcement Learning from Human Feedback (RLHF) methods to make them better
aligned with users' preferences. In contrast to LLMs, human preference learning
has not been widely explored in text-to-image diffusion models; the best
existing approach is to fine-tune a pretrained model using carefully curated
high quality images and captions to improve visual appeal and text alignment.
We propose Diffusion-DPO, a method to align diffusion models to human
preferences by directly optimizing on human comparison data. Diffusion-DPO is
adapted from the recently developed Direct Preference Optimization (DPO), a
simpler alternative to RLHF which directly optimizes a policy that best
satisfies human preferences under a classification objective. We re-formulate
DPO to account for a diffusion model notion of likelihood, utilizing the
evidence lower bound to derive a differentiable objective. Using the Pick-a-Pic
dataset of 851K crowdsourced pairwise preferences, we fine-tune the base model
of the state-of-the-art Stable Diffusion XL (SDXL)-1.0 model with
Diffusion-DPO. Our fine-tuned base model significantly outperforms both base
SDXL-1.0 and the larger SDXL-1.0 model consisting of an additional refinement
model in human evaluation, improving visual appeal and prompt alignment. We
also develop a variant that uses AI feedback and has comparable performance to
training on human preferences, opening the door for scaling of diffusion model
alignment methods.
A method called Diffusion-DPO aligns text-to-image diffusion models to human preferences using direct optimization on comparison data, improving visual appeal and prompt alignment.
Reinforcement Learning from Human Feedback (RLHF) human comparison data diffusion models human preference learning text-to-image diffusion models Direct Preference Optimization (DPO) evidence lower bound Pick-a-Pic dataset Stable Diffusion XL (SDXL)-1.0 AI feedback