RLPR: Extrapolating RLVR to General Domains without Verifiers
Reinforcement Learning with Verifiable Rewards (RLVR) demonstrates promising
potential in advancing the reasoning capabilities of LLMs. However, its success
remains largely confined to mathematical and code domains. This primary
limitation stems from the heavy reliance on domain-specific verifiers, which
results in prohibitive complexity and limited scalability. To address the
challenge, our key observation is that LLM's intrinsic probability of
generating a correct free-form answer directly indicates its own evaluation of
the reasoning reward (i.e., how well the reasoning process leads to the correct
answer). Building on this insight, we propose RLPR, a simple verifier-free
framework that extrapolates RLVR to broader general domains. RLPR uses the
LLM's own token probability scores for reference answers as the reward signal
and maximizes the expected reward during training. We find that addressing the
high variance of this noisy probability reward is crucial to make it work, and
propose prob-to-reward and stabilizing methods to ensure a precise and stable
reward from LLM intrinsic probabilities. Comprehensive experiments in four
general-domain benchmarks and three mathematical benchmarks show that RLPR
consistently improves reasoning capabilities in both areas for Gemma, Llama,
and Qwen based models. Notably, RLPR outperforms concurrent VeriFree by 7.6
points on TheoremQA and 7.5 points on Minerva, and even surpasses strong
verifier-model-dependent approaches General-Reasoner by 1.6 average points
across seven benchmarks.
685a39d60e4ad7e219758622
Tianyu Yu
claimed_verified
2025-06-24T12:43:15.822Z
64abc4aa6cadc7aca585dddf
/avatars/736afea979cd0021c7a37f68731524ea.svg
Tianyu Yu
user
Yirany
685a39d60e4ad7e219758623
Bo Ji
685a39d60e4ad7e219758624
Shouli Wang
685a39d60e4ad7e219758625
Shu Yao
685a39d60e4ad7e219758626
Zefan Wang
685a39d60e4ad7e219758627
Ganqu Cui
685a39d60e4ad7e219758628
Lifan Yuan
685a39d60e4ad7e219758629
Ning Ding
claimed_verified
2025-09-13T15:04:56.298Z
60cf4bcb1ce3775ebb86e5d5
/avatars/12bcd18d215abf91f297f93007733148.svg
Ning Ding
user
stingning
685a39d60e4ad7e21975862a
Yuan Yao
685a39d60e4ad7e21975862b
Zhiyuan Liu
685a39d60e4ad7e21975862c
Maosong Sun
685a39d60e4ad7e21975862d
Tat-Seng Chua
admin_assigned
2025-06-24T13:41:22.951Z
6570ae84c4993b8fb96f41a8
/avatars/21f7d79d46ac4df0ecff8eca7678b33f.svg
Tat-Seng Chua
user
chuats
RLPR: Extrapolating RLVR to General Domains without Verifiers
Reinforcement Learning with Verifiable Rewards (RLVR) demonstrates promising
potential in advancing the reasoning capabilities of LLMs. However, its success
remains largely confined to mathematical and code domains. This primary
limitation stems from the heavy reliance on domain-specific verifiers, which
results in prohibitive complexity and limited scalability. To address the
challenge, our key observation is that LLM's intrinsic probability of
generating a correct free-form answer directly indicates its own evaluation of
the reasoning reward (i.e., how well the reasoning process leads to the correct
answer). Building on this insight, we propose RLPR, a simple verifier-free
framework that extrapolates RLVR to broader general domains. RLPR uses the
LLM's own token probability scores for reference answers as the reward signal
and maximizes the expected reward during training. We find that addressing the
high variance of this noisy probability reward is crucial to make it work, and
propose prob-to-reward and stabilizing methods to ensure a precise and stable
reward from LLM intrinsic probabilities. Comprehensive experiments in four
general-domain benchmarks and three mathematical benchmarks show that RLPR
consistently improves reasoning capabilities in both areas for Gemma, Llama,
and Qwen based models. Notably, RLPR outperforms concurrent VeriFree by 7.6
points on TheoremQA and 7.5 points on Minerva, and even surpasses strong
verifier-model-dependent approaches General-Reasoner by 1.6 average points
across seven benchmarks.
RLPR, a verifier-free framework using LLM's token probability scores as reward signals, enhances reasoning capabilities across both general and mathematical domains, outperforming other methods in various benchmarks.
Reinforcement Learning with Verifiable Rewards (RLVR) reasoning capabilities LLM RLPR token probability scores prob-to-reward stabilizing methods TheoremQA Minerva General-Reasoner