Reinforcing Diffusion Models by Direct Group Preference Optimization
While reinforcement learning methods such as Group Relative Preference
Optimization (GRPO) have significantly enhanced Large Language Models, adapting
them to diffusion models remains challenging. In particular, GRPO demands a
stochastic policy, yet the most cost-effective diffusion samplers are based on
deterministic ODEs. Recent work addresses this issue by using inefficient
SDE-based samplers to induce stochasticity, but this reliance on model-agnostic
Gaussian noise leads to slow convergence. To resolve this conflict, we propose
Direct Group Preference Optimization (DGPO), a new online RL algorithm that
dispenses with the policy-gradient framework entirely. DGPO learns directly
from group-level preferences, which utilize relative information of samples
within groups. This design eliminates the need for inefficient stochastic
policies, unlocking the use of efficient deterministic ODE samplers and faster
training. Extensive results show that DGPO trains around 20 times faster than
existing state-of-the-art methods and achieves superior performance on both
in-domain and out-of-domain reward metrics. Code is available at
https://github.com/Luo-Yihong/DGPO.
68e874b595e8e6771df38a02
Yihong Luo
claimed_verified
2025-10-15T15:25:36.835Z
65f7e6856bd4bac5b6a4ecc3
/avatars/9c0c48310fb5e99c19700316a0ab531e.svg
Yihong Luo
user
Luo-Yihong
68e874b595e8e6771df38a03
Tianyang Hu
68e874b595e8e6771df38a04
Jing Tang
Reinforcing Diffusion Models by Direct Group Preference Optimization
While reinforcement learning methods such as Group Relative Preference
Optimization (GRPO) have significantly enhanced Large Language Models, adapting
them to diffusion models remains challenging. In particular, GRPO demands a
stochastic policy, yet the most cost-effective diffusion samplers are based on
deterministic ODEs. Recent work addresses this issue by using inefficient
SDE-based samplers to induce stochasticity, but this reliance on model-agnostic
Gaussian noise leads to slow convergence. To resolve this conflict, we propose
Direct Group Preference Optimization (DGPO), a new online RL algorithm that
dispenses with the policy-gradient framework entirely. DGPO learns directly
from group-level preferences, which utilize relative information of samples
within groups. This design eliminates the need for inefficient stochastic
policies, unlocking the use of efficient deterministic ODE samplers and faster
training. Extensive results show that DGPO trains around 20 times faster than
existing state-of-the-art methods and achieves superior performance on both
in-domain and out-of-domain reward metrics. Code is available at
https://github.com/Luo-Yihong/DGPO.
DGPO, a new online RL algorithm, enhances diffusion models by learning from group-level preferences, enabling the use of efficient deterministic ODE samplers and achieving faster training and superior performance.
Group Relative Preference Optimization GRPO diffusion models stochastic policy deterministic ODEs SDE-based samplers Gaussian noise Direct Group Preference Optimization DGPO policy-gradient framework group-level preferences in-domain reward metrics out-of-domain reward metrics