SePPO: Semi-Policy Preference Optimization for Diffusion Alignment
Reinforcement learning from human feedback (RLHF) methods are emerging as a
way to fine-tune diffusion models (DMs) for visual generation. However,
commonly used on-policy strategies are limited by the generalization capability
of the reward model, while off-policy approaches require large amounts of
difficult-to-obtain paired human-annotated data, particularly in visual
generation tasks. To address the limitations of both on- and off-policy RLHF,
we propose a preference optimization method that aligns DMs with preferences
without relying on reward models or paired human-annotated data. Specifically,
we introduce a Semi-Policy Preference Optimization (SePPO) method. SePPO
leverages previous checkpoints as reference models while using them to generate
on-policy reference samples, which replace "losing images" in preference pairs.
This approach allows us to optimize using only off-policy "winning images."
Furthermore, we design a strategy for reference model selection that expands
the exploration in the policy space. Notably, we do not simply treat reference
samples as negative examples for learning. Instead, we design an anchor-based
criterion to assess whether the reference samples are likely to be winning or
losing images, allowing the model to selectively learn from the generated
reference samples. This approach mitigates performance degradation caused by
the uncertainty in reference sample quality. We validate SePPO across both
text-to-image and text-to-video benchmarks. SePPO surpasses all previous
approaches on the text-to-image benchmarks and also demonstrates outstanding
performance on the text-to-video benchmarks. Code will be released in
https://github.com/DwanZhang-AI/SePPO.
670540d5ff94857c894b1595
Daoan Zhang
admin_assigned
2024-10-08T15:37:15.699Z
6241b95cfee9374a2598ecfe
/avatars/196669df1689a5872fc18b271e80fdc1.svg
Zhang Daoan
user
hazard
670540d5ff94857c894b1596
Guangchen Lan
claimed_verified
2024-10-09T07:37:54.329Z
64ff4b1a0e8369f6a8c47c7e
https://cdn-avatars.huggingface.co/v1/production/uploads/64ff4b1a0e8369f6a8c47c7e/X8ocOagj89gexSa253q_8.png
Eric Lan
user
Eric-Lan
670540d5ff94857c894b1597
Dong-Jun Han
670540d5ff94857c894b1598
Wenlin Yao
admin_assigned
2024-10-08T15:37:40.892Z
634f18e4aae4bde2c8e2adca
/avatars/40549a59fc5ba04a4baa5a1d5dba0847.svg
Wenlin Yao
user
wenlinyao
670540d5ff94857c894b1599
Xiaoman Pan
670540d5ff94857c894b159a
Hongming Zhang
admin_assigned
2024-10-08T15:38:12.792Z
64ed478bec06efeb03034933
/avatars/cd7dc3165831e90cb36d39d41c3c8157.svg
Hongming Zhang
user
Hongming98
670540d5ff94857c894b159b
Mingxiao Li
admin_assigned
2024-10-08T15:38:21.097Z
63457bb1abd499c80af15dcc
/avatars/dddfdc51a50f48d97c4308b2c169433c.svg
MingxiaoLi
user
Lixiaoming
670540d5ff94857c894b159c
Pengcheng Chen
670540d5ff94857c894b159d
Yu Dong
670540d5ff94857c894b159e
Christopher Brinton
670540d5ff94857c894b159f
Jiebo Luo
SePPO: Semi-Policy Preference Optimization for Diffusion Alignment
Reinforcement learning from human feedback (RLHF) methods are emerging as a
way to fine-tune diffusion models (DMs) for visual generation. However,
commonly used on-policy strategies are limited by the generalization capability
of the reward model, while off-policy approaches require large amounts of
difficult-to-obtain paired human-annotated data, particularly in visual
generation tasks. To address the limitations of both on- and off-policy RLHF,
we propose a preference optimization method that aligns DMs with preferences
without relying on reward models or paired human-annotated data. Specifically,
we introduce a Semi-Policy Preference Optimization (SePPO) method. SePPO
leverages previous checkpoints as reference models while using them to generate
on-policy reference samples, which replace "losing images" in preference pairs.
This approach allows us to optimize using only off-policy "winning images."
Furthermore, we design a strategy for reference model selection that expands
the exploration in the policy space. Notably, we do not simply treat reference
samples as negative examples for learning. Instead, we design an anchor-based
criterion to assess whether the reference samples are likely to be winning or
losing images, allowing the model to selectively learn from the generated
reference samples. This approach mitigates performance degradation caused by
the uncertainty in reference sample quality. We validate SePPO across both
text-to-image and text-to-video benchmarks. SePPO surpasses all previous
approaches on the text-to-image benchmarks and also demonstrates outstanding
performance on the text-to-video benchmarks. Code will be released in
https://github.com/DwanZhang-AI/SePPO.
A novel preference optimization method, SePPO, aligns diffusion models with human preferences without reward models or paired data, demonstrating superior performance in text-to-image and text-to-video tasks.
Reinforcement learning from human feedback (RLHF) diffusion models (DMs) on-policy strategies off-policy approaches reward models paired human-annotated data preference optimization Semi-Policy Preference Optimization (SePPO) reference models on-policy reference samples anchor-based criterion policy space performance degradation text-to-image text-to-video