Aligning Diffusion Models by Optimizing Human Utility
We present Diffusion-KTO, a novel approach for aligning text-to-image
diffusion models by formulating the alignment objective as the maximization of
expected human utility. Since this objective applies to each generation
independently, Diffusion-KTO does not require collecting costly pairwise
preference data nor training a complex reward model. Instead, our objective
requires simple per-image binary feedback signals, e.g. likes or dislikes,
which are abundantly available. After fine-tuning using Diffusion-KTO,
text-to-image diffusion models exhibit superior performance compared to
existing techniques, including supervised fine-tuning and Diffusion-DPO, both
in terms of human judgment and automatic evaluation metrics such as PickScore
and ImageReward. Overall, Diffusion-KTO unlocks the potential of leveraging
readily available per-image binary signals and broadens the applicability of
aligning text-to-image diffusion models with human preferences.
6614c4b0c13e5d702f5b6ca2
Shufan Li
claimed_verified
2024-12-04T09:47:40.616Z
6310531914aa81e1044363ed
/avatars/ae7767e591cb7199ea2f62d2db89fc7f.svg
Shufan Li
user
jacklishufan
6614c4b0c13e5d702f5b6ca3
Konstantinos Kallidromitis
6614c4b0c13e5d702f5b6ca4
Akash Gokul
6614c4b0c13e5d702f5b6ca5
Yusuke Kato
6614c4b0c13e5d702f5b6ca6
Kazuki Kozuka
Aligning Diffusion Models by Optimizing Human Utility
We present Diffusion-KTO, a novel approach for aligning text-to-image
diffusion models by formulating the alignment objective as the maximization of
expected human utility. Since this objective applies to each generation
independently, Diffusion-KTO does not require collecting costly pairwise
preference data nor training a complex reward model. Instead, our objective
requires simple per-image binary feedback signals, e.g. likes or dislikes,
which are abundantly available. After fine-tuning using Diffusion-KTO,
text-to-image diffusion models exhibit superior performance compared to
existing techniques, including supervised fine-tuning and Diffusion-DPO, both
in terms of human judgment and automatic evaluation metrics such as PickScore
and ImageReward. Overall, Diffusion-KTO unlocks the potential of leveraging
readily available per-image binary signals and broadens the applicability of
aligning text-to-image diffusion models with human preferences.
Diffusion-KTO fine-tunes text-to-image diffusion models using per-image binary feedback, enhancing performance in both human and automatic evaluations.
Diffusion-KTO text-to-image diffusion models alignment objective human utility pairwise preference data reward model per-image binary feedback signals supervised fine-tuning Diffusion-DPO PickScore ImageReward