DiffusionNFT: Online Diffusion Reinforcement with Forward Process
Online reinforcement learning (RL) has been central to post-training language
models, but its extension to diffusion models remains challenging due to
intractable likelihoods. Recent works discretize the reverse sampling process
to enable GRPO-style training, yet they inherit fundamental drawbacks,
including solver restrictions, forward-reverse inconsistency, and complicated
integration with classifier-free guidance (CFG). We introduce Diffusion
Negative-aware FineTuning (DiffusionNFT), a new online RL paradigm that
optimizes diffusion models directly on the forward process via flow matching.
DiffusionNFT contrasts positive and negative generations to define an implicit
policy improvement direction, naturally incorporating reinforcement signals
into the supervised learning objective. This formulation enables training with
arbitrary black-box solvers, eliminates the need for likelihood estimation, and
requires only clean images rather than sampling trajectories for policy
optimization. DiffusionNFT is up to 25times more efficient than FlowGRPO in
head-to-head comparisons, while being CFG-free. For instance, DiffusionNFT
improves the GenEval score from 0.24 to 0.98 within 1k steps, while FlowGRPO
achieves 0.95 with over 5k steps and additional CFG employment. By leveraging
multiple reward models, DiffusionNFT significantly boosts the performance of
SD3.5-Medium in every benchmark tested.
68d0bc4f8adc5cd018d15afb
Kaiwen Zheng
admin_assigned
2025-09-23T14:12:30.463Z
652bf7edc3cba555d5673c6e
/avatars/78f6416c30203b30671f8423f061c657.svg
Kaiwen Zheng
user
worstcoder
68d0bc4f8adc5cd018d15afc
Huayu Chen
claimed_verified
2025-09-23T15:15:47.039Z
65571135bfb62d747abc8129
/avatars/5f4542daa34597f17e6280b9cce18c91.svg
Hugging
user
ChenDRAG
68d0bc4f8adc5cd018d15afd
Haotian Ye
68d0bc4f8adc5cd018d15afe
Haoxiang Wang
68d0bc4f8adc5cd018d15aff
Qinsheng Zhang
admin_assigned
2025-09-23T14:12:52.095Z
6732d5dea24987c43bfbafd8
/avatars/1581373b9de5069975716932fceb976b.svg
Qinsheng Zhang
user
qsh-zh
68d0bc4f8adc5cd018d15b00
Kai Jiang
68d0bc4f8adc5cd018d15b01
Hang Su
68d0bc4f8adc5cd018d15b02
Stefano Ermon
admin_assigned
2025-09-23T14:12:59.507Z
62f6e244329d4d014d1f4ac5
/avatars/5a8b2bb063c2ebc340504b22530f6811.svg
Stefano Ermon
user
ermonste
68d0bc4f8adc5cd018d15b03
Jun Zhu
68d0bc4f8adc5cd018d15b04
Ming-Yu Liu
admin_assigned
2025-09-23T14:13:08.835Z
62f049afdf4b93aad5c7f2d6
/avatars/e272e58ad996733d7098e50248e5b57e.svg
Ming-Yu Liu
user
mingyuliutw
DiffusionNFT: Online Diffusion Reinforcement with Forward Process
Online reinforcement learning (RL) has been central to post-training language
models, but its extension to diffusion models remains challenging due to
intractable likelihoods. Recent works discretize the reverse sampling process
to enable GRPO-style training, yet they inherit fundamental drawbacks,
including solver restrictions, forward-reverse inconsistency, and complicated
integration with classifier-free guidance (CFG). We introduce Diffusion
Negative-aware FineTuning (DiffusionNFT), a new online RL paradigm that
optimizes diffusion models directly on the forward process via flow matching.
DiffusionNFT contrasts positive and negative generations to define an implicit
policy improvement direction, naturally incorporating reinforcement signals
into the supervised learning objective. This formulation enables training with
arbitrary black-box solvers, eliminates the need for likelihood estimation, and
requires only clean images rather than sampling trajectories for policy
optimization. DiffusionNFT is up to 25times more efficient than FlowGRPO in
head-to-head comparisons, while being CFG-free. For instance, DiffusionNFT
improves the GenEval score from 0.24 to 0.98 within 1k steps, while FlowGRPO
achieves 0.95 with over 5k steps and additional CFG employment. By leveraging
multiple reward models, DiffusionNFT significantly boosts the performance of
SD3.5-Medium in every benchmark tested.
Diffusion Negative-aware FineTuning (DiffusionNFT) optimizes diffusion models directly on the forward process via flow matching, improving efficiency and performance compared to existing methods.
online reinforcement learning diffusion models GRPO-style training reverse sampling process flow matching implicit policy improvement supervised learning objective black-box solvers likelihood estimation GenEval score reward models SD3.5-Medium