Scalable Ranked Preference Optimization for Text-to-Image Generation
Direct Preference Optimization (DPO) has emerged as a powerful approach to
align text-to-image (T2I) models with human feedback. Unfortunately, successful
application of DPO to T2I models requires a huge amount of resources to collect
and label large-scale datasets, e.g., millions of generated paired images
annotated with human preferences. In addition, these human preference datasets
can get outdated quickly as the rapid improvements of T2I models lead to higher
quality images. In this work, we investigate a scalable approach for collecting
large-scale and fully synthetic datasets for DPO training. Specifically, the
preferences for paired images are generated using a pre-trained reward
function, eliminating the need for involving humans in the annotation process,
greatly improving the dataset collection efficiency. Moreover, we demonstrate
that such datasets allow averaging predictions across multiple models and
collecting ranked preferences as opposed to pairwise preferences. Furthermore,
we introduce RankDPO to enhance DPO-based methods using the ranking feedback.
Applying RankDPO on SDXL and SD3-Medium models with our synthetically generated
preference dataset ``Syn-Pic'' improves both prompt-following (on benchmarks
like T2I-Compbench, GenEval, and DPG-Bench) and visual quality (through user
studies). This pipeline presents a practical and scalable solution to develop
better preference datasets to enhance the performance of text-to-image models.
671a01b4c673da43e82b2867
Shyamgopal Karthik
claimed_verified
2024-10-24T08:31:42.279Z
6254599b6e36fe62e141c8f9
/avatars/08d6b68b92c7bfd0a8be022ba9f2f289.svg
Shyamgopal Karthik
user
shyamgopal
671a01b4c673da43e82b2868
Huseyin Coskun
admin_assigned
2024-10-24T08:35:13.318Z
64eaefcd5349043b2b9670a2
/avatars/9ca92fdda3f7b14586f40e9d31a746fe.svg
Huseyin Coskun
user
hcoskun
671a01b4c673da43e82b2869
Zeynep Akata
671a01b4c673da43e82b286a
Sergey Tulyakov
671a01b4c673da43e82b286b
Jian Ren
claimed_verified
2025-01-15T08:49:46.801Z
61f19829233c91cbd2f79e70
/avatars/a0735a94542b4f7cda5aed8bc4be0538.svg
Jian Ren
user
alanspike
671a01b4c673da43e82b286c
Anil Kag
admin_assigned
2024-10-24T08:35:38.040Z
66b01ee8e53bbad918362856
/avatars/293529589a91dd7a95909d66727db224.svg
Anil Kag
user
anilkagak2
Scalable Ranked Preference Optimization for Text-to-Image Generation
Direct Preference Optimization (DPO) has emerged as a powerful approach to
align text-to-image (T2I) models with human feedback. Unfortunately, successful
application of DPO to T2I models requires a huge amount of resources to collect
and label large-scale datasets, e.g., millions of generated paired images
annotated with human preferences. In addition, these human preference datasets
can get outdated quickly as the rapid improvements of T2I models lead to higher
quality images. In this work, we investigate a scalable approach for collecting
large-scale and fully synthetic datasets for DPO training. Specifically, the
preferences for paired images are generated using a pre-trained reward
function, eliminating the need for involving humans in the annotation process,
greatly improving the dataset collection efficiency. Moreover, we demonstrate
that such datasets allow averaging predictions across multiple models and
collecting ranked preferences as opposed to pairwise preferences. Furthermore,
we introduce RankDPO to enhance DPO-based methods using the ranking feedback.
Applying RankDPO on SDXL and SD3-Medium models with our synthetically generated
preference dataset ``Syn-Pic'' improves both prompt-following (on benchmarks
like T2I-Compbench, GenEval, and DPG-Bench) and visual quality (through user
studies). This pipeline presents a practical and scalable solution to develop
better preference datasets to enhance the performance of text-to-image models.
A scalable approach using a pre-trained reward function generates synthetic preference datasets for Direct Preference Optimization (DPO) in text-to-image models, improving prompt-following and visual quality.
Direct Preference Optimization (DPO) text-to-image (T2I) pre-trained reward function synthetic datasets ranking feedback RankDPO T2I-Compbench GenEval DPG-Bench