RewardDance: Reward Scaling in Visual Generation
Reward Models (RMs) are critical for improving generation models via
Reinforcement Learning (RL), yet the RM scaling paradigm in visual generation
remains largely unexplored. It primarily due to fundamental limitations in
existing approaches: CLIP-based RMs suffer from architectural and input
modality constraints, while prevalent Bradley-Terry losses are fundamentally
misaligned with the next-token prediction mechanism of Vision-Language Models
(VLMs), hindering effective scaling. More critically, the RLHF optimization
process is plagued by Reward Hacking issue, where models exploit flaws in the
reward signal without improving true quality. To address these challenges, we
introduce RewardDance, a scalable reward modeling framework that overcomes
these barriers through a novel generative reward paradigm. By reformulating the
reward score as the model's probability of predicting a "yes" token, indicating
that the generated image outperforms a reference image according to specific
criteria, RewardDance intrinsically aligns reward objectives with VLM
architectures. This alignment unlocks scaling across two dimensions: (1) Model
Scaling: Systematic scaling of RMs up to 26 billion parameters; (2) Context
Scaling: Integration of task-specific instructions, reference examples, and
chain-of-thought (CoT) reasoning. Extensive experiments demonstrate that
RewardDance significantly surpasses state-of-the-art methods in text-to-image,
text-to-video, and image-to-video generation. Crucially, we resolve the
persistent challenge of "reward hacking": Our large-scale RMs exhibit and
maintain high reward variance during RL fine-tuning, proving their resistance
to hacking and ability to produce diverse, high-quality outputs. It greatly
relieves the mode collapse problem that plagues smaller models.
68c2308c29b8ec9932cd08f3
Jie Wu
claimed_verified
2025-09-12T16:14:15.007Z
6381c5d63680a7cf34e08ca9
/avatars/731467e2d80d0ae163c4a00a9e3ff9e5.svg
wujie10558@gmail.com
user
wujie10
68c2308c29b8ec9932cd08f4
Yu Gao
68c2308c29b8ec9932cd08f5
Zilyu Ye
68c2308c29b8ec9932cd08f6
Ming Li
claimed_verified
2025-09-12T16:14:11.142Z
637f0eb22438d7485b8ef5d7
https://cdn-avatars.huggingface.co/v1/production/uploads/637f0eb22438d7485b8ef5d7/70h7dekqj7LuBobOXckmJ.jpeg
Ming Li
user
limingcv
68c2308c29b8ec9932cd08f7
Liang Li
68c2308c29b8ec9932cd08f8
Hanzhong Guo
68c2308c29b8ec9932cd08f9
Jie Liu
claimed_verified
2025-09-24T13:57:37.217Z
639be86b59473c6ae02ef9c4
https://cdn-avatars.huggingface.co/v1/production/uploads/639be86b59473c6ae02ef9c4/gw34RBCVZCOkcAA79xUr3.png
Jie Liu
user
jieliu
68c2308c29b8ec9932cd08fa
Zeyue Xue
68c2308c29b8ec9932cd08fb
Xiaoxia Hou
68c2308c29b8ec9932cd08fc
Wei Liu
68c2308c29b8ec9932cd08fd
Yan Zeng
68c2308c29b8ec9932cd08fe
Weilin Huang
RewardDance: Reward Scaling in Visual Generation
Reward Models (RMs) are critical for improving generation models via
Reinforcement Learning (RL), yet the RM scaling paradigm in visual generation
remains largely unexplored. It primarily due to fundamental limitations in
existing approaches: CLIP-based RMs suffer from architectural and input
modality constraints, while prevalent Bradley-Terry losses are fundamentally
misaligned with the next-token prediction mechanism of Vision-Language Models
(VLMs), hindering effective scaling. More critically, the RLHF optimization
process is plagued by Reward Hacking issue, where models exploit flaws in the
reward signal without improving true quality. To address these challenges, we
introduce RewardDance, a scalable reward modeling framework that overcomes
these barriers through a novel generative reward paradigm. By reformulating the
reward score as the model's probability of predicting a "yes" token, indicating
that the generated image outperforms a reference image according to specific
criteria, RewardDance intrinsically aligns reward objectives with VLM
architectures. This alignment unlocks scaling across two dimensions: (1) Model
Scaling: Systematic scaling of RMs up to 26 billion parameters; (2) Context
Scaling: Integration of task-specific instructions, reference examples, and
chain-of-thought (CoT) reasoning. Extensive experiments demonstrate that
RewardDance significantly surpasses state-of-the-art methods in text-to-image,
text-to-video, and image-to-video generation. Crucially, we resolve the
persistent challenge of "reward hacking": Our large-scale RMs exhibit and
maintain high reward variance during RL fine-tuning, proving their resistance
to hacking and ability to produce diverse, high-quality outputs. It greatly
relieves the mode collapse problem that plagues smaller models.
RewardDance is a scalable reward modeling framework that aligns with VLM architectures, enabling effective scaling of RMs and resolving reward hacking issues in generation models.
CLIP-based RMs Bradley-Terry losses Vision-Language Models (VLMs) RLHF optimization Reward Hacking RewardDance generative reward paradigm model scaling context scaling task-specific instructions reference examples chain-of-thought (CoT) reasoning text-to-image text-to-video image-to-video generation mode collapse