RLFR: Extending Reinforcement Learning for LLMs with Flow Environment
Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as
a promising framework for improving reasoning abilities in Large Language
Models (LLMs). However, policy optimized with binary verification prone to
overlook potential valuable exploration in reasoning trajectory. In view of
heavy annotation cost of golden Process Reward Models (PRMs), recent works
attempt using auxiliary signals for reward shaping of process tokens, involving
entropy and likelihood collected from logit space. In this work, we offer a
novel perspective on shaping RLVR with flow rewards derived from latent space,
and propose RLFR, where the flow fields of model latents are constructed from
either off-policy high-quality data and on-policy rejection sampling data, and
the velocity deviations of policy latents within it are quantified to serve as
a reward signal. RLFR first demonstrates that a well-established flow field can
be a sound environment for reward signal collection, highlighting the
expressive latent space is much underexplored. Moreover, RLFR is able to
compress any off-policy expert data as reference for constituting reward
signals, and we show that the efficient context dependence compressed within
the hidden states are utilized, rather than individual token-level denotation
for context comprehending. Experiments on both language and multimodal
reasoning benchmarks demonstrate the reliability of flow rewards, and
suggesting a promising paradigm for reward shaping with auxiliary signals.
68edab32de1fee572713a6eb
Jinghao Zhang
claimed_verified
2025-10-14T07:31:05.248Z
64673258fc6f6da8b119cab8
/avatars/36e025862984c7a86b97cee750ee2d04.svg
SII-Jhao Zhang
user
JingHaoZ
68edab32de1fee572713a6ec
Naishan Zheng
68edab32de1fee572713a6ed
Ruilin Li
68edab32de1fee572713a6ee
Dongzhou Cheng
68edab32de1fee572713a6ef
Zheming Liang
68edab32de1fee572713a6f0
Feng Zhao
68edab32de1fee572713a6f1
Jiaqi Wang
RLFR: Extending Reinforcement Learning for LLMs with Flow Environment
Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as
a promising framework for improving reasoning abilities in Large Language
Models (LLMs). However, policy optimized with binary verification prone to
overlook potential valuable exploration in reasoning trajectory. In view of
heavy annotation cost of golden Process Reward Models (PRMs), recent works
attempt using auxiliary signals for reward shaping of process tokens, involving
entropy and likelihood collected from logit space. In this work, we offer a
novel perspective on shaping RLVR with flow rewards derived from latent space,
and propose RLFR, where the flow fields of model latents are constructed from
either off-policy high-quality data and on-policy rejection sampling data, and
the velocity deviations of policy latents within it are quantified to serve as
a reward signal. RLFR first demonstrates that a well-established flow field can
be a sound environment for reward signal collection, highlighting the
expressive latent space is much underexplored. Moreover, RLFR is able to
compress any off-policy expert data as reference for constituting reward
signals, and we show that the efficient context dependence compressed within
the hidden states are utilized, rather than individual token-level denotation
for context comprehending. Experiments on both language and multimodal
reasoning benchmarks demonstrate the reliability of flow rewards, and
suggesting a promising paradigm for reward shaping with auxiliary signals.
RLFR uses flow rewards derived from latent space to improve reinforcement learning with verifiable rewards, demonstrating reliable reward shaping and efficient context comprehension.
Reinforcement Learning with Verifiable Rewards RLVR Large Language Models LLMs Process Reward Models PRMs reward shaping process tokens logit space flow rewards latent space flow fields off-policy data on-policy rejection sampling velocity deviations hidden states context comprehension language benchmarks multimodal reasoning benchmarks