ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning
Tonghe Zhang, Chao Yu, Sichang Su, Yu Wang
We propose ReinFlow, a simple and effective online reinforcement learning (RL) framework that fine-tunes a family of flow matching policies for continuous robotic control. ReinFlow injects a bounded, learnable noise into a flow policy's deterministic path, converting the flow into a discrete-time Markov Process for simple and tractable likelihood computation. This conversion aids exploration and secures training stability, allowing ReinFlow to stably fine-tune diverse flow model variants, including Rectified Flow and Shortcut Models, especially at very few or even one denoising step. We benchmark ReinFlow in representative locomotion and manipulation tasks, including long-horizon planning with visual input and sparse reward. The episode reward of Rectified Flow policies increased by an average of 162.28\% after fine-tuning in challenging legged locomotion tasks, while saving 82.87\% of wall-time compared to the state-of-the-art diffusion RL method DPPO. The success rate of the Shortcut Model policies in state and visual manipulation tasks increased by 39.86\% on average after fine-tuning with ReinFlow at four or even one denoising step, achieving performance comparable to fine-tuned DDIM policies while saving 48.78\% of the simulation time.