G^2RPO: Granular GRPO for Precise Reward in Flow Models
The integration of online reinforcement learning (RL) into diffusion and flow
models has recently emerged as a promising approach for aligning generative
models with human preferences. Stochastic sampling via Stochastic Differential
Equations (SDE) is employed during the denoising process to generate diverse
denoising directions for RL exploration. While existing methods effectively
explore potential high-value samples, they suffer from sub-optimal preference
alignment due to sparse and narrow reward signals. To address these challenges,
we propose a novel Granular-GRPO (G^2RPO ) framework that achieves
precise and comprehensive reward assessments of sampling directions in
reinforcement learning of flow models. Specifically, a Singular Stochastic
Sampling strategy is introduced to support step-wise stochastic exploration
while enforcing a high correlation between the reward and the injected noise,
thereby facilitating a faithful reward for each SDE perturbation. Concurrently,
to eliminate the bias inherent in fixed-granularity denoising, we introduce a
Multi-Granularity Advantage Integration module that aggregates advantages
computed at multiple diffusion scales, producing a more comprehensive and
robust evaluation of the sampling directions. Experiments conducted on various
reward models, including both in-domain and out-of-domain evaluations,
demonstrate that our G^2RPO significantly outperforms existing
flow-based GRPO baselines,highlighting its effectiveness and robustness.
68e73f567ae125f9582e6a2c
Yujie Zhou
claimed_verified
2025-10-11T13:48:52.696Z
64b51fd8bcfd8542d6473d9a
/avatars/ceaa73b79f448996187f07733d96b800.svg
yujie
user
yujieouo
68e73f567ae125f9582e6a2d
Pengyang Ling
68e73f567ae125f9582e6a2e
Jiazi Bu
68e73f567ae125f9582e6a2f
Yibin Wang
claimed_verified
2025-10-22T15:46:47.718Z
654c6845bac6e6e49895a5b5
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/KXQaAxulqr8jNBSpEaYM4.png
SII-Yibin Wang
user
CodeGoat24
68e73f567ae125f9582e6a30
Yuhang Zang
claimed_verified
2025-10-09T06:49:50.572Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/kajwuVzd4pDucSPlwghxo.png
Yuhang Zang
user
yuhangzang
68e73f567ae125f9582e6a31
Jiaqi Wang
68e73f567ae125f9582e6a32
Li Niu
68e73f567ae125f9582e6a33
Guangtao Zhai
G^2RPO: Granular GRPO for Precise Reward in Flow Models
The integration of online reinforcement learning (RL) into diffusion and flow
models has recently emerged as a promising approach for aligning generative
models with human preferences. Stochastic sampling via Stochastic Differential
Equations (SDE) is employed during the denoising process to generate diverse
denoising directions for RL exploration. While existing methods effectively
explore potential high-value samples, they suffer from sub-optimal preference
alignment due to sparse and narrow reward signals. To address these challenges,
we propose a novel Granular-GRPO (G^2RPO ) framework that achieves
precise and comprehensive reward assessments of sampling directions in
reinforcement learning of flow models. Specifically, a Singular Stochastic
Sampling strategy is introduced to support step-wise stochastic exploration
while enforcing a high correlation between the reward and the injected noise,
thereby facilitating a faithful reward for each SDE perturbation. Concurrently,
to eliminate the bias inherent in fixed-granularity denoising, we introduce a
Multi-Granularity Advantage Integration module that aggregates advantages
computed at multiple diffusion scales, producing a more comprehensive and
robust evaluation of the sampling directions. Experiments conducted on various
reward models, including both in-domain and out-of-domain evaluations,
demonstrate that our G^2RPO significantly outperforms existing
flow-based GRPO baselines,highlighting its effectiveness and robustness.
A novel Granular-GRPO framework enhances reinforcement learning in diffusion and flow models by improving reward assessment and reducing bias in denoising.
online reinforcement learning diffusion models flow models Stochastic Differential Equations SDE denoising process Singular Stochastic Sampling Multi-Granularity Advantage Integration reward signals reward assessment sampling directions in-domain evaluations out-of-domain evaluations GRPO