Flow-GRPO: Training Flow Matching Models via Online RL
We propose Flow-GRPO, the first method integrating online reinforcement
learning (RL) into flow matching models. Our approach uses two key strategies:
(1) an ODE-to-SDE conversion that transforms a deterministic Ordinary
Differential Equation (ODE) into an equivalent Stochastic Differential Equation
(SDE) that matches the original model's marginal distribution at all timesteps,
enabling statistical sampling for RL exploration; and (2) a Denoising Reduction
strategy that reduces training denoising steps while retaining the original
inference timestep number, significantly improving sampling efficiency without
performance degradation. Empirically, Flow-GRPO is effective across multiple
text-to-image tasks. For complex compositions, RL-tuned SD3.5 generates nearly
perfect object counts, spatial relations, and fine-grained attributes, boosting
GenEval accuracy from 63% to 95%. In visual text rendering, its accuracy
improves from 59% to 92%, significantly enhancing text generation.
Flow-GRPO also achieves substantial gains in human preference alignment.
Notably, little to no reward hacking occurred, meaning rewards did not increase
at the cost of image quality or diversity, and both remained stable in our
experiments.
681d9829edf34a77aab565eb
Jie Liu
claimed_verified
2025-05-09T13:54:46.048Z
639be86b59473c6ae02ef9c4
https://cdn-avatars.huggingface.co/v1/production/uploads/639be86b59473c6ae02ef9c4/gw34RBCVZCOkcAA79xUr3.png
Jie Liu
user
jieliu
681d9829edf34a77aab565ec
Gongye Liu
admin_assigned
2025-05-09T08:41:47.403Z
6553316bf151de82f6a23e1d
https://cdn-avatars.huggingface.co/v1/production/uploads/6553316bf151de82f6a23e1d/GTBkSj4Fa3OoyM6Muz_Sc.jpeg
Gongye Liu
user
liuhuohuo
681d9829edf34a77aab565ed
Jiajun Liang
681d9829edf34a77aab565ee
Yangguang Li
admin_assigned
2025-05-09T08:43:44.697Z
64d71083a787c9bc7b9f1238
/avatars/d0b0546dec7fc5792921154bec41385a.svg
Yangguang Li
user
Lp256
681d9829edf34a77aab565ef
Jiaheng Liu
admin_assigned
2025-05-09T08:45:07.297Z
65377c30e48353201e6fdda0
/avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
Jiaheng Liu
user
CheeryLJH
681d9829edf34a77aab565f0
Xintao Wang
admin_assigned
2025-05-09T08:42:27.977Z
60e272ca6c78a8c122b12127
https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
Xintao Wang
user
Xintao
681d9829edf34a77aab565f1
Pengfei Wan
681d9829edf34a77aab565f2
Di Zhang
admin_assigned
2025-05-09T08:43:01.366Z
644c8324f02250233d0d67d9
/avatars/feb39d281457c1750f3eada3c060a23e.svg
Di Zhang
user
dizhang
681d9829edf34a77aab565f3
Wanli Ouyang
Flow-GRPO: Training Flow Matching Models via Online RL
We propose Flow-GRPO, the first method integrating online reinforcement
learning (RL) into flow matching models. Our approach uses two key strategies:
(1) an ODE-to-SDE conversion that transforms a deterministic Ordinary
Differential Equation (ODE) into an equivalent Stochastic Differential Equation
(SDE) that matches the original model's marginal distribution at all timesteps,
enabling statistical sampling for RL exploration; and (2) a Denoising Reduction
strategy that reduces training denoising steps while retaining the original
inference timestep number, significantly improving sampling efficiency without
performance degradation. Empirically, Flow-GRPO is effective across multiple
text-to-image tasks. For complex compositions, RL-tuned SD3.5 generates nearly
perfect object counts, spatial relations, and fine-grained attributes, boosting
GenEval accuracy from 63% to 95%. In visual text rendering, its accuracy
improves from 59% to 92%, significantly enhancing text generation.
Flow-GRPO also achieves substantial gains in human preference alignment.
Notably, little to no reward hacking occurred, meaning rewards did not increase
at the cost of image quality or diversity, and both remained stable in our
experiments.
Flow-GRPO combines online reinforcement learning with flow matching models through an ODE-to-SDE conversion and denoising reduction, improving sampling efficiency and performance across text-to-image tasks.
Flow-GRPO online reinforcement learning flow matching models ODE-to-SDE conversion Stochastic Differential Equation Denoising Reduction text-to-image tasks RL-tuned SD3.5 GenEval visual text rendering human preference alignment