Free Lunch Alignment of Text-to-Image Diffusion Models without
  Preference Image Pairs
Recent advances in diffusion-based text-to-image (T2I) models have led to
remarkable success in generating high-quality images from textual prompts.
However, ensuring accurate alignment between the text and the generated image
remains a significant challenge for state-of-the-art diffusion models. To
address this, existing studies employ reinforcement learning with human
feedback (RLHF) to align T2I outputs with human preferences. These methods,
however, either rely directly on paired image preference data or require a
learned reward function, both of which depend heavily on costly, high-quality
human annotations and thus face scalability limitations. In this work, we
introduce Text Preference Optimization (TPO), a framework that enables
"free-lunch" alignment of T2I models, achieving alignment without the need for
paired image preference data. TPO works by training the model to prefer matched
prompts over mismatched prompts, which are constructed by perturbing original
captions using a large language model. Our framework is general and compatible
with existing preference-based algorithms. We extend both DPO and KTO to our
setting, resulting in TDPO and TKTO. Quantitative and qualitative evaluations
across multiple benchmarks show that our methods consistently outperform their
original counterparts, delivering better human preference scores and improved
text-to-image alignment. Our Open-source code is available at
https://github.com/DSL-Lab/T2I-Free-Lunch-Alignment.
68e336df73e20ab577842024
Jia Jun Cheng Xian
68e336df73e20ab577842025
Muchen Li
68e336df73e20ab577842026
Haotian Yang
68e336df73e20ab577842027
Xin Tao
68e336df73e20ab577842028
Pengfei Wan
68e336df73e20ab577842029
Leonid Sigal
68e336df73e20ab57784202a
Renjie Liao
Free Lunch Alignment of Text-to-Image Diffusion Models without
  Preference Image Pairs
Recent advances in diffusion-based text-to-image (T2I) models have led to
remarkable success in generating high-quality images from textual prompts.
However, ensuring accurate alignment between the text and the generated image
remains a significant challenge for state-of-the-art diffusion models. To
address this, existing studies employ reinforcement learning with human
feedback (RLHF) to align T2I outputs with human preferences. These methods,
however, either rely directly on paired image preference data or require a
learned reward function, both of which depend heavily on costly, high-quality
human annotations and thus face scalability limitations. In this work, we
introduce Text Preference Optimization (TPO), a framework that enables
"free-lunch" alignment of T2I models, achieving alignment without the need for
paired image preference data. TPO works by training the model to prefer matched
prompts over mismatched prompts, which are constructed by perturbing original
captions using a large language model. Our framework is general and compatible
with existing preference-based algorithms. We extend both DPO and KTO to our
setting, resulting in TDPO and TKTO. Quantitative and qualitative evaluations
across multiple benchmarks show that our methods consistently outperform their
original counterparts, delivering better human preference scores and improved
text-to-image alignment. Our Open-source code is available at
https://github.com/DSL-Lab/T2I-Free-Lunch-Alignment.
A new framework, Text Preference Optimization (TPO), aligns text-to-image models with human preferences without requiring paired image preference data, improving text-to-image alignment and human preference scores.
diffusion-based text-to-image T2I models reinforcement learning with human feedback RLHF Text Preference Optimization TPO matched prompts mismatched prompts large language model DPO KTO TDPO TKTO