FlowRL: Matching Reward Distributions for LLM Reasoning
We propose FlowRL: matching the full reward distribution via flow balancing
instead of maximizing rewards in large language model (LLM) reinforcement
learning (RL). Recent advanced reasoning models adopt reward-maximizing methods
(\eg, PPO and GRPO), which tend to over-optimize dominant reward signals while
neglecting less frequent but valid reasoning paths, thus reducing diversity. In
contrast, we transform scalar rewards into a normalized target distribution
using a learnable partition function, and then minimize the reverse KL
divergence between the policy and the target distribution. We implement this
idea as a flow-balanced optimization method that promotes diverse exploration
and generalizable reasoning trajectories. We conduct experiments on math and
code reasoning tasks: FlowRL achieves a significant average improvement of
10.0% over GRPO and 5.1% over PPO on math benchmarks, and performs
consistently better on code reasoning tasks. These results highlight reward
distribution-matching as a key step toward efficient exploration and diverse
reasoning in LLM reinforcement learning.
68ccb7983df9ac65e93dc626
Xuekai Zhu
claimed_verified
2025-09-19T06:48:45.314Z
647ffddeb82adfa7cc1a10d9
/avatars/26aa168d6b2068298ebb16584aa52b6c.svg
zhu
user
xuekai
68ccb7983df9ac65e93dc627
Daixuan Cheng
claimed_verified
2025-09-19T06:48:48.340Z
649e6761f9134a06ed1e0cea
https://cdn-avatars.huggingface.co/v1/production/uploads/649e6761f9134a06ed1e0cea/XNeKceE8xSwI0xWwWUwwJ.jpeg
Daixuan Cheng
user
daixuancheng
68ccb7983df9ac65e93dc628
Dinghuai Zhang
admin_assigned
2025-09-19T13:11:24.546Z
64384e18d221ff12edae4c75
/avatars/3a5d2400af0f26091c233d63984df412.svg
Dinghuai Zhang
user
Dinghuai
68ccb7983df9ac65e93dc629
Hengli Li
68ccb7983df9ac65e93dc62a
Kaiyan Zhang
68ccb7983df9ac65e93dc62b
Che Jiang
68ccb7983df9ac65e93dc62c
Youbang Sun
68ccb7983df9ac65e93dc62d
Ermo Hua
68ccb7983df9ac65e93dc62e
Yuxin Zuo
admin_assigned
2025-09-19T13:11:54.405Z
622474f38dc6b0b64f5e903d
/avatars/d6b60a014277a8ec7d564163c5f644aa.svg
Yuxin Zuo
user
yuxinzuo
68ccb7983df9ac65e93dc62f
Xingtai Lv
admin_assigned
2025-09-19T13:11:46.119Z
663f07d029be04778ba97871
/avatars/fb7c9d4a2c537d918a3267e7cbc03f04.svg
Xingtai Lv
user
XingtaiHF
68ccb7983df9ac65e93dc630
Qizheng Zhang
admin_assigned
2025-09-19T13:12:02.816Z
6663e5b10c54dffcd2a921ca
/avatars/2a4589fef05306ccde06728c752e5601.svg
Qizheng Zhang
user
qizhengz
68ccb7983df9ac65e93dc631
Lin Chen
claimed_verified
2025-09-21T13:13:01.825Z
67c27673e5911f17dbdded18
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/hp1VNuQe9Lf8QH0-km5dQ.png
LChen
user
Charlie-LChen
68ccb7983df9ac65e93dc632
Fanghao Shao
68ccb7983df9ac65e93dc633
Bo Xue
68ccb7983df9ac65e93dc634
Yunchong Song
68ccb7983df9ac65e93dc635
Zhenjie Yang
claimed_verified
2025-09-19T06:48:37.248Z
65574c70d16b524f1d2345c1
/avatars/8edbe9ebea85c570bc19b63bf3a727d9.svg
Zhenjie Yang
user
jayyoung0802
68ccb7983df9ac65e93dc636
Ganqu Cui
admin_assigned
2025-09-19T13:12:20.955Z
650eba9555dc1e841746f132
/avatars/af6f5ee78f161d25ec0afc45d2def8eb.svg
Ganqu Cui
user
ganqu
68ccb7983df9ac65e93dc637
Ning Ding
68ccb7983df9ac65e93dc638
Jianfeng Gao
admin_assigned
2025-09-19T13:12:29.442Z
641904caf9d6f1d772ec7af7
/avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
Jianfeng Gao
user
wyngjf
68ccb7983df9ac65e93dc639
Xiaodong Liu
68ccb7983df9ac65e93dc63a
Bowen Zhou
68ccb7983df9ac65e93dc63b
Hongyuan Mei
68ccb7983df9ac65e93dc63c
Zhouhan Lin
claimed_verified
2025-09-23T10:07:24.087Z
68d21f8ca090df93e72c09a0
/avatars/3aac21cea09bf30230bab4d52dfc0582.svg
Zhouhan Lin
user
ZhouhanLin
FlowRL: Matching Reward Distributions for LLM Reasoning
We propose FlowRL: matching the full reward distribution via flow balancing
instead of maximizing rewards in large language model (LLM) reinforcement
learning (RL). Recent advanced reasoning models adopt reward-maximizing methods
(\eg, PPO and GRPO), which tend to over-optimize dominant reward signals while
neglecting less frequent but valid reasoning paths, thus reducing diversity. In
contrast, we transform scalar rewards into a normalized target distribution
using a learnable partition function, and then minimize the reverse KL
divergence between the policy and the target distribution. We implement this
idea as a flow-balanced optimization method that promotes diverse exploration
and generalizable reasoning trajectories. We conduct experiments on math and
code reasoning tasks: FlowRL achieves a significant average improvement of
10.0% over GRPO and 5.1% over PPO on math benchmarks, and performs
consistently better on code reasoning tasks. These results highlight reward
distribution-matching as a key step toward efficient exploration and diverse
reasoning in LLM reinforcement learning.
FlowRL enhances LLM reinforcement learning by matching the full reward distribution through flow balancing, improving diversity and performance over reward-maximizing methods.
FlowRL reward distribution flow balancing reinforcement learning reward-maximizing methods PPO GRPO normalized target distribution learnable partition function reverse KL divergence diverse exploration generalizable reasoning trajectories math reasoning code reasoning