Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable
  Text-to-Image Reinforcement Learning
Recent advancements highlight the importance of GRPO-based reinforcement
learning methods and benchmarking in enhancing text-to-image (T2I) generation.
However, current methods using pointwise reward models (RM) for scoring
generated images are susceptible to reward hacking. We reveal that this happens
when minimal score differences between images are amplified after
normalization, creating illusory advantages that drive the model to
over-optimize for trivial gains, ultimately destabilizing the image generation
process. To address this, we propose Pref-GRPO, a pairwise preference
reward-based GRPO method that shifts the optimization objective from score
maximization to preference fitting, ensuring more stable training. In
Pref-GRPO, images are pairwise compared within each group using preference RM,
and the win rate is used as the reward signal. Extensive experiments
demonstrate that PREF-GRPO differentiates subtle image quality differences,
providing more stable advantages and mitigating reward hacking. Additionally,
existing T2I benchmarks are limited by coarse evaluation criteria, hindering
comprehensive model assessment. To solve this, we introduce UniGenBench, a
unified T2I benchmark comprising 600 prompts across 5 main themes and 20
subthemes. It evaluates semantic consistency through 10 primary and 27
sub-criteria, leveraging MLLM for benchmark construction and evaluation. Our
benchmarks uncover the strengths and weaknesses of both open and closed-source
T2I models and validate the effectiveness of Pref-GRPO.
68b10e71b19c54000148491a
Yibin Wang
claimed_verified
2025-08-29T08:02:30.189Z
654c6845bac6e6e49895a5b5
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/KXQaAxulqr8jNBSpEaYM4.png
SII-Yibin Wang
user
CodeGoat24
68b10e71b19c54000148491b
Zhimin Li
68b10e71b19c54000148491c
Yuhang Zang
claimed_verified
2025-08-29T08:02:32.217Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/kajwuVzd4pDucSPlwghxo.png
Yuhang Zang
user
yuhangzang
68b10e71b19c54000148491d
Yujie Zhou
68b10e71b19c54000148491e
Jiazi Bu
68b10e71b19c54000148491f
Chunyu Wang
68b10e71b19c540001484920
Qinglin Lu
68b10e71b19c540001484921
Cheng Jin
68b10e71b19c540001484922
Jiaqi Wang
Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable
  Text-to-Image Reinforcement Learning
Recent advancements highlight the importance of GRPO-based reinforcement
learning methods and benchmarking in enhancing text-to-image (T2I) generation.
However, current methods using pointwise reward models (RM) for scoring
generated images are susceptible to reward hacking. We reveal that this happens
when minimal score differences between images are amplified after
normalization, creating illusory advantages that drive the model to
over-optimize for trivial gains, ultimately destabilizing the image generation
process. To address this, we propose Pref-GRPO, a pairwise preference
reward-based GRPO method that shifts the optimization objective from score
maximization to preference fitting, ensuring more stable training. In
Pref-GRPO, images are pairwise compared within each group using preference RM,
and the win rate is used as the reward signal. Extensive experiments
demonstrate that PREF-GRPO differentiates subtle image quality differences,
providing more stable advantages and mitigating reward hacking. Additionally,
existing T2I benchmarks are limited by coarse evaluation criteria, hindering
comprehensive model assessment. To solve this, we introduce UniGenBench, a
unified T2I benchmark comprising 600 prompts across 5 main themes and 20
subthemes. It evaluates semantic consistency through 10 primary and 27
sub-criteria, leveraging MLLM for benchmark construction and evaluation. Our
benchmarks uncover the strengths and weaknesses of both open and closed-source
T2I models and validate the effectiveness of Pref-GRPO.
Pref-GRPO, a pairwise preference reward-based GRPO method, enhances text-to-image generation by mitigating reward hacking and improving stability, while UniGenBench provides a comprehensive benchmark for evaluating T2I models.
GRPO reinforcement learning text-to-image pointwise reward models reward hacking pairwise preference preference fitting win rate UniGenBench semantic consistency MLLM