S^2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement
  Learning
Recent studies have demonstrated the effectiveness of LLM test-time scaling.
However, existing approaches to incentivize LLMs' deep thinking abilities
generally require large-scale data or significant training efforts. Meanwhile,
it remains unclear how to improve the thinking abilities of less powerful base
models. In this work, we introduce S^2R, an efficient framework that enhances
LLM reasoning by teaching models to self-verify and self-correct during
inference. Specifically, we first initialize LLMs with iterative
self-verification and self-correction behaviors through supervised fine-tuning
on carefully curated data. The self-verification and self-correction skills are
then further strengthened by both outcome-level and process-level reinforcement
learning, with minimized resource requirements, enabling the model to
adaptively refine its reasoning process during inference. Our results
demonstrate that, with only 3.1k self-verifying and self-correcting behavior
initialization samples, Qwen2.5-math-7B achieves an accuracy improvement from
51.0\% to 81.6\%, outperforming models trained on an equivalent amount of
long-CoT distilled data. Extensive experiments and analysis based on three base
models across both in-domain and out-of-domain benchmarks validate the
effectiveness of S^2R. Our code and data are available at
https://github.com/NineAbyss/S2R.
67b69b6717ccb022c6a95b38
Ruotian Ma
admin_assigned
2025-02-21T14:58:55.028Z
648294b2eb4befee378951c1
/avatars/da5d8bf9d8662cc2ffa2c0de49bd66a3.svg
Ruotian Ma
user
vvibt
67b69b6717ccb022c6a95b39
Peisong Wang
admin_assigned
2025-02-21T14:59:00.273Z
626f98528a894872cfbf620c
/avatars/fe31d20313e6ca85e96bc249424c5383.svg
Peisong Wang
user
duke1852022
67b69b6717ccb022c6a95b3a
Cheng Liu
claimed_verified
2025-02-24T09:25:27.094Z
6500234be0c94282ab38cd00
/avatars/90fc160919cdbb28cfa82becf720b062.svg
soso
user
chengliu
67b69b6717ccb022c6a95b3b
Xingyan Liu
67b69b6717ccb022c6a95b3c
Jiaqi Chen
claimed_verified
2025-04-29T07:59:24.130Z
61b859ddbdf1fac5ed499992
/avatars/2387fb9b8a46840bfc75248462f0a410.svg
Jiaqi Chen
user
judge
67b69b6717ccb022c6a95b3d
Bang Zhang
67b69b6717ccb022c6a95b3e
Xin Zhou
67b69b6717ccb022c6a95b3f
Nan Du
67b69b6717ccb022c6a95b40
Jia Li
S^2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement
  Learning
Recent studies have demonstrated the effectiveness of LLM test-time scaling.
However, existing approaches to incentivize LLMs' deep thinking abilities
generally require large-scale data or significant training efforts. Meanwhile,
it remains unclear how to improve the thinking abilities of less powerful base
models. In this work, we introduce S^2R, an efficient framework that enhances
LLM reasoning by teaching models to self-verify and self-correct during
inference. Specifically, we first initialize LLMs with iterative
self-verification and self-correction behaviors through supervised fine-tuning
on carefully curated data. The self-verification and self-correction skills are
then further strengthened by both outcome-level and process-level reinforcement
learning, with minimized resource requirements, enabling the model to
adaptively refine its reasoning process during inference. Our results
demonstrate that, with only 3.1k self-verifying and self-correcting behavior
initialization samples, Qwen2.5-math-7B achieves an accuracy improvement from
51.0\% to 81.6\%, outperforming models trained on an equivalent amount of
long-CoT distilled data. Extensive experiments and analysis based on three base
models across both in-domain and out-of-domain benchmarks validate the
effectiveness of S^2R. Our code and data are available at
https://github.com/NineAbyss/S2R.
S$^2$R is a framework that enhances LLM reasoning through iterative self-verification and self-correction using supervised fine-tuning and reinforcement learning with minimal resources.
LLM self-verification self-correction supervised fine-tuning reinforcement learning Qwen2.5-math-7B long-CoT in-domain out-of-domain benchmarks