SPRO: Improving Image Generation via Self-Play
Ritika Jha, Aanisha Bhattacharyya, Yaman Singla, Rajiv Ratn Shah, Changyou Chen, Balaji Krishnamurthy
Recent advances in diffusion models have led to impressive improvements in image fidelity and diversity. However, aligning these models with nuanced human preferences such as aesthetic quality, engagement, and subjective user appeal remains a major challenge in the absence of large-scale human annotations. Collecting data that captures such preferences is both expensive and limited in diversity. To address this, we leverage the reasoning capabilities of Vision-language models (VLMs) and introduce a scalable, human-annotation-free training paradigm based on self-play. We present Self-Play Reward Optimization (SPRO), a unified, multimodal, multi-model framework that aligns both prompt generation and image synthesis with human preferences using synthetic reward signals. SPRO operates in three stages: SPRO-Prompt trains a Guider-VLM through self-play to generate diverse, high-reward prompts tailored to objectives such as PickScore (user preference), LAION-Aesthetics, and EngagetNet (engagement); SPRO-Image fine-tunes the diffusion model using high-reward images produced in the previous stage; and SPRO-Multimodal (SPRO-MM) combines both components, using optimized prompts and a fine-tuned generator for full-capacity alignment. Without relying on human-annotated data, SPRO achieves an average 30\% improvement across objectives and generalizes across both open-source and proprietary diffusion models. Additionally, SPRO uncovers novel prompting strategies rarely authored by humans, such as visual harmony for aesthetics and background or shadow-based cues for engagement. Across a range of diffusion pipelines, SPRO significantly outperforms prior methods and commercial baselines, demonstrating a scalable path toward aligning generative models with diverse human preferences.