NOVER: Incentive Training for Language Models via Verifier-Free
  Reinforcement Learning
Recent advances such as DeepSeek R1-Zero highlight the effectiveness of
incentive training, a reinforcement learning paradigm that computes rewards
solely based on the final answer part of a language model's output, thereby
encouraging the generation of intermediate reasoning steps. However, these
methods fundamentally rely on external verifiers, which limits their
applicability to domains like mathematics and coding where such verifiers are
readily available. Although reward models can serve as verifiers, they require
high-quality annotated data and are costly to train. In this work, we propose
NOVER, NO-VERifier Reinforcement Learning, a general reinforcement learning
framework that requires only standard supervised fine-tuning data with no need
for an external verifier. NOVER enables incentive training across a wide range
of text-to-text tasks and outperforms the model of the same size distilled from
large reasoning models such as DeepSeek R1 671B by 7.7 percent. Moreover, the
flexibility of NOVER enables new possibilities for optimizing large language
models, such as inverse incentive training.
68342cb2924393051af84722
Wei Liu
claimed_verified
2025-05-26T14:07:36.205Z
66e2932e5c100c12aa2def39
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/FiQ5Fap-qVqnXeULGPYs6.png
weiliu
user
thinkwee
68342cb2924393051af84723
Siya Qi
68342cb2924393051af84724
Xinyu Wang
68342cb2924393051af84725
Chen Qian
68342cb2924393051af84726
Yali Du
68342cb2924393051af84727
Yulan He
NOVER: Incentive Training for Language Models via Verifier-Free
  Reinforcement Learning
Recent advances such as DeepSeek R1-Zero highlight the effectiveness of
incentive training, a reinforcement learning paradigm that computes rewards
solely based on the final answer part of a language model's output, thereby
encouraging the generation of intermediate reasoning steps. However, these
methods fundamentally rely on external verifiers, which limits their
applicability to domains like mathematics and coding where such verifiers are
readily available. Although reward models can serve as verifiers, they require
high-quality annotated data and are costly to train. In this work, we propose
NOVER, NO-VERifier Reinforcement Learning, a general reinforcement learning
framework that requires only standard supervised fine-tuning data with no need
for an external verifier. NOVER enables incentive training across a wide range
of text-to-text tasks and outperforms the model of the same size distilled from
large reasoning models such as DeepSeek R1 671B by 7.7 percent. Moreover, the
flexibility of NOVER enables new possibilities for optimizing large language
models, such as inverse incentive training.
NOVER, a reinforcement learning framework that eliminates the need for external verifiers, enhances language model performance across text-to-text tasks.
incentive training reinforcement learning NOVER NO-VERifier Reinforcement Learning DeepSeek R1-Zero DeepSeek R1 671B inverse incentive training