PVPO: Pre-Estimated Value-Based Policy Optimization for Agentic
  Reasoning
Critic-free reinforcement learning methods, particularly group policies, have
attracted considerable attention for their efficiency in complex tasks.
However, these methods rely heavily on multiple sampling and comparisons within
the policy to estimate advantage, which may cause the policy to fall into local
optimum and increase computational cost. To address these issues, we propose
PVPO, an efficient reinforcement learning method enhanced by an advantage
reference anchor and data pre-sampling. Specifically, we use the reference
model to rollout in advance and employ the calculated reward score as a
reference anchor. Our approach effectively corrects the cumulative bias
introduced by intra-group comparisons and significantly reduces reliance on the
number of rollouts. Meanwhile, the reference model can assess sample difficulty
during data pre-sampling, enabling effective selection of high-gain data to
improve training efficiency. Experiments conducted on nine datasets across two
domains demonstrate that PVPO achieves State-Of-The-Art (SOTA) performance. Our
approach not only demonstrates robust generalization across multiple tasks, but
also exhibits scalable performance across models of varying scales.
68b50650851c6e7b001eca07
Wenfeng Feng
claimed_verified
2025-09-04T12:51:39.425Z
68b691341c52b8b741ba7fd4
/avatars/efae3c632577565eb3645e64d4f725ac.svg
wenfeng feng
user
wenfengfwf
68b50650851c6e7b001eca08
Penghong Zhao
admin_assigned
2025-09-09T16:13:16.697Z
66e2a76a420a3111a8b6acb8
/avatars/858810348c289f4b1937315f9cc15779.svg
Penghong Zhao
user
DDDDrop
68b50650851c6e7b001eca09
Guochao Jiang
claimed_verified
2025-09-01T07:50:45.446Z
644a1dbb9c340e5e1e713153
/avatars/21cb93ad067a798a39829ef7e67c70b8.svg
JGC
user
Nothing2Say
68b50650851c6e7b001eca0a
Chuzhan Hao
claimed_verified
2025-09-02T07:57:08.870Z
64ae631b58bd9e9cc2f5a749
/avatars/ce6426ec3bdb618a9e449297e7f147e0.svg
Chuzhan HAO
user
Chuzhan
68b50650851c6e7b001eca0b
Yuewei Zhang
claimed_verified
2025-09-03T08:29:36.912Z
68b6ec9b1224e891a08873bd
/avatars/58604eab370fe35ee1053a1c33874982.svg
yuewei zhang
user
zhangywlfh
68b50650851c6e7b001eca0c
Hao Wang
PVPO: Pre-Estimated Value-Based Policy Optimization for Agentic
  Reasoning
Critic-free reinforcement learning methods, particularly group policies, have
attracted considerable attention for their efficiency in complex tasks.
However, these methods rely heavily on multiple sampling and comparisons within
the policy to estimate advantage, which may cause the policy to fall into local
optimum and increase computational cost. To address these issues, we propose
PVPO, an efficient reinforcement learning method enhanced by an advantage
reference anchor and data pre-sampling. Specifically, we use the reference
model to rollout in advance and employ the calculated reward score as a
reference anchor. Our approach effectively corrects the cumulative bias
introduced by intra-group comparisons and significantly reduces reliance on the
number of rollouts. Meanwhile, the reference model can assess sample difficulty
during data pre-sampling, enabling effective selection of high-gain data to
improve training efficiency. Experiments conducted on nine datasets across two
domains demonstrate that PVPO achieves State-Of-The-Art (SOTA) performance. Our
approach not only demonstrates robust generalization across multiple tasks, but
also exhibits scalable performance across models of varying scales.
PVPO, an enhanced reinforcement learning method using a reference anchor and data pre-sampling, achieves state-of-the-art performance with reduced computational cost and improved generalization.
critic-free reinforcement learning group policies advantage estimation local optimum computational cost advantage reference anchor data pre-sampling reference model rollout reward score cumulative bias intra-group comparisons sample difficulty high-gain data training efficiency state-of-the-art performance robust generalization scalable performance