Reinforcing General Reasoning without Verifiers
The recent paradigm shift towards training large language models (LLMs) using
DeepSeek-R1-Zero-style reinforcement learning (RL) on verifiable rewards has
led to impressive advancements in code and mathematical reasoning. However,
this methodology is limited to tasks where rule-based answer verification is
possible and does not naturally extend to real-world domains such as chemistry,
healthcare, engineering, law, biology, business, and economics. Current
practical workarounds use an additional LLM as a model-based verifier; however,
this introduces issues such as reliance on a strong verifier LLM,
susceptibility to reward hacking, and the practical burden of maintaining the
verifier model in memory during training. To address this and extend
DeepSeek-R1-Zero-style training to general reasoning domains, we propose a
verifier-free method (VeriFree) that bypasses answer verification and instead
uses RL to directly maximize the probability of generating the reference
answer. We compare VeriFree with verifier-based methods and demonstrate that,
in addition to its significant practical benefits and reduced compute
requirements, VeriFree matches and even surpasses verifier-based methods on
extensive evaluations across MMLU-Pro, GPQA, SuperGPQA, and math-related
benchmarks. Moreover, we provide insights into this method from multiple
perspectives: as an elegant integration of training both the policy and
implicit verifier in a unified model, and as a variational optimization
approach. Code is available at https://github.com/sail-sg/VeriFree.
68372228726e45ea3f19fd59
Xiangxin Zhou
claimed_verified
2025-06-02T07:48:45.621Z
66129c7b50350afe76757262
/avatars/a2f4fac076b9d658a0d904ed54960f6f.svg
Xiangxin Zhou
user
zhouxiangxin
68372228726e45ea3f19fd5a
Zichen Liu
claimed_verified
2025-05-28T19:13:48.854Z
65f5392c68b8e0cb3c9977a2
/avatars/aa64772475098e8a135c13072fde6744.svg
Zichen
user
lkevinzc
68372228726e45ea3f19fd5b
Anya Sims
68372228726e45ea3f19fd5c
Haonan Wang
68372228726e45ea3f19fd5d
Tianyu Pang
68372228726e45ea3f19fd5e
Chongxuan Li
68372228726e45ea3f19fd5f
Liang Wang
68372228726e45ea3f19fd60
Min Lin
68372228726e45ea3f19fd61
Chao Du
Reinforcing General Reasoning without Verifiers
The recent paradigm shift towards training large language models (LLMs) using
DeepSeek-R1-Zero-style reinforcement learning (RL) on verifiable rewards has
led to impressive advancements in code and mathematical reasoning. However,
this methodology is limited to tasks where rule-based answer verification is
possible and does not naturally extend to real-world domains such as chemistry,
healthcare, engineering, law, biology, business, and economics. Current
practical workarounds use an additional LLM as a model-based verifier; however,
this introduces issues such as reliance on a strong verifier LLM,
susceptibility to reward hacking, and the practical burden of maintaining the
verifier model in memory during training. To address this and extend
DeepSeek-R1-Zero-style training to general reasoning domains, we propose a
verifier-free method (VeriFree) that bypasses answer verification and instead
uses RL to directly maximize the probability of generating the reference
answer. We compare VeriFree with verifier-based methods and demonstrate that,
in addition to its significant practical benefits and reduced compute
requirements, VeriFree matches and even surpasses verifier-based methods on
extensive evaluations across MMLU-Pro, GPQA, SuperGPQA, and math-related
benchmarks. Moreover, we provide insights into this method from multiple
perspectives: as an elegant integration of training both the policy and
implicit verifier in a unified model, and as a variational optimization
approach. Code is available at https://github.com/sail-sg/VeriFree.
A verifier-free method (VeriFree) is introduced to extend reinforcement learning training of large language models to general reasoning domains, improving efficiency and performance compared to verifier-based methods.
DeepSeek-R1-Zero reinforcement learning RL large language models LLMs VeriFree MMLU-Pro GPQA SuperGPQA