Random Policy Valuation is Enough for LLM Reasoning with Verifiable
  Rewards
RL with Verifiable Rewards (RLVR) has emerged as a promising paradigm for
improving the reasoning abilities of large language models (LLMs). Current
methods rely primarily on policy optimization frameworks like PPO and GRPO,
which follow generalized policy iteration that alternates between evaluating
the current policy's value and improving the policy based on evaluation. While
effective, they often suffer from training instability and diversity collapse,
requiring complex heuristic tricks and careful tuning. We observe that standard
RLVR in math reasoning can be formalized as a specialized finite-horizon Markov
Decision Process with deterministic state transitions, tree-structured
dynamics, and binary terminal rewards. Though large in scale, the underlying
structure is simpler than general-purpose control settings for which popular RL
algorithms (e.g., PPO) were developed, suggesting that several sophisticated
techniques in existing methods may be reduced or even omitted. Based on this
insight, we prove a surprising result: the optimal action can be recovered from
the Q-function of a fixed uniformly random policy, thereby bypassing the
generalized policy iteration loop and its associated heuristics. We introduce
Random Policy Valuation for Diverse Reasoning (ROVER) to translate this
principle into a practical and scalable algorithm for LLM math reasoning, a
minimalist yet highly effective RL method that samples actions from a softmax
over these uniform-policy Q-values. ROVER preserves diversity throughout
training, allowing sustained exploration of multiple valid pathways. Across
multiple base models and standard math reasoning benchmarks, ROVER demonstrates
superior performance in both quality (+8.2 on pass@1,
+16.8 on pass@256) and diversity (+17.6\%), despite
its radical simplification compared to strong, complicated existing methods.
68db71fbd2bf1f4b15ec76e3
Haoran He
68db71fbd2bf1f4b15ec76e4
Yuxiao Ye
68db71fbd2bf1f4b15ec76e5
Qingpeng Cai
68db71fbd2bf1f4b15ec76e6
Chen Hu
claimed_verified
2025-10-10T09:56:36.997Z
648183bfea00b120773198ba
/avatars/b69d04bba11e393d1a59e98653d5cfa6.svg
huchen
user
ccchen1006
68db71fbd2bf1f4b15ec76e7
Binxing Jiao
68db71fbd2bf1f4b15ec76e8
Daxin Jiang
68db71fbd2bf1f4b15ec76e9
Ling Pan
Random Policy Valuation is Enough for LLM Reasoning with Verifiable
  Rewards
RL with Verifiable Rewards (RLVR) has emerged as a promising paradigm for
improving the reasoning abilities of large language models (LLMs). Current
methods rely primarily on policy optimization frameworks like PPO and GRPO,
which follow generalized policy iteration that alternates between evaluating
the current policy's value and improving the policy based on evaluation. While
effective, they often suffer from training instability and diversity collapse,
requiring complex heuristic tricks and careful tuning. We observe that standard
RLVR in math reasoning can be formalized as a specialized finite-horizon Markov
Decision Process with deterministic state transitions, tree-structured
dynamics, and binary terminal rewards. Though large in scale, the underlying
structure is simpler than general-purpose control settings for which popular RL
algorithms (e.g., PPO) were developed, suggesting that several sophisticated
techniques in existing methods may be reduced or even omitted. Based on this
insight, we prove a surprising result: the optimal action can be recovered from
the Q-function of a fixed uniformly random policy, thereby bypassing the
generalized policy iteration loop and its associated heuristics. We introduce
Random Policy Valuation for Diverse Reasoning (ROVER) to translate this
principle into a practical and scalable algorithm for LLM math reasoning, a
minimalist yet highly effective RL method that samples actions from a softmax
over these uniform-policy Q-values. ROVER preserves diversity throughout
training, allowing sustained exploration of multiple valid pathways. Across
multiple base models and standard math reasoning benchmarks, ROVER demonstrates
superior performance in both quality (+8.2 on pass@1,
+16.8 on pass@256) and diversity (+17.6\%), despite
its radical simplification compared to strong, complicated existing methods.
ROVER, a minimalist RL method, achieves superior performance and diversity in LLM math reasoning by leveraging Q-values from a fixed random policy, bypassing complex policy iteration.
RL with Verifiable Rewards RLVR large language models LLMs policy optimization PPO GRPO generalized policy iteration Markov Decision Process deterministic state transitions tree-structured dynamics binary terminal rewards Q-function Random Policy Valuation for Diverse Reasoning ROVER softmax exploration pass@1 pass@256 diversity