MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE
Although GRPO substantially enhances flow matching models in human preference
alignment of image generation, methods such as FlowGRPO still exhibit
inefficiency due to the necessity of sampling and optimizing over all denoising
steps specified by the Markov Decision Process (MDP). In this paper, we propose
MixGRPO, a novel framework that leverages the flexibility of mixed
sampling strategies through the integration of stochastic differential
equations (SDE) and ordinary differential equations (ODE). This streamlines the
optimization process within the MDP to improve efficiency and boost
performance. Specifically, MixGRPO introduces a sliding window mechanism, using
SDE sampling and GRPO-guided optimization only within the window, while
applying ODE sampling outside. This design confines sampling randomness to the
time-steps within the window, thereby reducing the optimization overhead, and
allowing for more focused gradient updates to accelerate convergence.
Additionally, as time-steps beyond the sliding window are not involved in
optimization, higher-order solvers are supported for sampling. So we present a
faster variant, termed MixGRPO-Flash, which further improves
training efficiency while achieving comparable performance. MixGRPO exhibits
substantial gains across multiple dimensions of human preference alignment,
outperforming DanceGRPO in both effectiveness and efficiency, with nearly 50%
lower training time. Notably, MixGRPO-Flash further reduces training time by
71%. Codes and models are available at
https://github.com/Tencent-Hunyuan/MixGRPO{MixGRPO}.
6889ce8231e1218a089282c1
Junzhe Li
claimed_verified
2025-07-31T21:01:07.591Z
64c860d23a3f428da65ea499
/avatars/f0bcc6ae7e558babe691b6bbf1059c9d.svg
lijunzhe
user
tulvgengenr
6889ce8231e1218a089282c2
Yutao Cui
6889ce8231e1218a089282c3
Tao Huang
6889ce8231e1218a089282c4
Yinping Ma
6889ce8231e1218a089282c5
Chun Fan
6889ce8231e1218a089282c6
Miles Yang
6889ce8231e1218a089282c7
Zhao Zhong
MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE
Although GRPO substantially enhances flow matching models in human preference
alignment of image generation, methods such as FlowGRPO still exhibit
inefficiency due to the necessity of sampling and optimizing over all denoising
steps specified by the Markov Decision Process (MDP). In this paper, we propose
MixGRPO, a novel framework that leverages the flexibility of mixed
sampling strategies through the integration of stochastic differential
equations (SDE) and ordinary differential equations (ODE). This streamlines the
optimization process within the MDP to improve efficiency and boost
performance. Specifically, MixGRPO introduces a sliding window mechanism, using
SDE sampling and GRPO-guided optimization only within the window, while
applying ODE sampling outside. This design confines sampling randomness to the
time-steps within the window, thereby reducing the optimization overhead, and
allowing for more focused gradient updates to accelerate convergence.
Additionally, as time-steps beyond the sliding window are not involved in
optimization, higher-order solvers are supported for sampling. So we present a
faster variant, termed MixGRPO-Flash, which further improves
training efficiency while achieving comparable performance. MixGRPO exhibits
substantial gains across multiple dimensions of human preference alignment,
outperforming DanceGRPO in both effectiveness and efficiency, with nearly 50%
lower training time. Notably, MixGRPO-Flash further reduces training time by
71%. Codes and models are available at
https://github.com/Tencent-Hunyuan/MixGRPO{MixGRPO}.
MixGRPO, a novel framework integrating SDE and ODE, enhances flow matching models for image generation by optimizing only within a sliding window, improving efficiency and performance.
flow matching models human preference alignment image generation Markov Decision Process stochastic differential equations ordinary differential equations sliding window mechanism gradient updates MixGRPO-Flash