Self-rewarding correction for mathematical reasoning
We study self-rewarding reasoning large language models (LLMs), which can
simultaneously generate step-by-step reasoning and evaluate the correctness of
their outputs during the inference time-without external feedback. This
integrated approach allows a single model to independently guide its reasoning
process, offering computational advantages for model deployment. We
particularly focus on the representative task of self-correction, where models
autonomously detect errors in their responses, revise outputs, and decide when
to terminate iterative refinement loops. To enable this, we propose a
two-staged algorithmic framework for constructing self-rewarding reasoning
models using only self-generated data. In the first stage, we employ sequential
rejection sampling to synthesize long chain-of-thought trajectories that
incorporate both self-rewarding and self-correction mechanisms. Fine-tuning
models on these curated data allows them to learn the patterns of
self-rewarding and self-correction. In the second stage, we further enhance the
models' ability to assess response accuracy and refine outputs through
reinforcement learning with rule-based signals. Experiments with Llama-3 and
Qwen-2.5 demonstrate that our approach surpasses intrinsic self-correction
capabilities and achieves performance comparable to systems that rely on
external reward models.
67c12987505a88e4a185e0d7
Wei Xiong
67c12987505a88e4a185e0d8
Hanning Zhang
admin_assigned
2025-02-28T12:22:33.128Z
6470e0f1cfd57849519033a5
/avatars/7ffefee3e36a4e37b9f4510bc6b689d1.svg
Hanning Zhang
user
HanningZhang
67c12987505a88e4a185e0d9
Chenlu Ye
admin_assigned
2025-02-28T12:22:38.981Z
65eec5c1d7d63c2ed0615421
/avatars/8c32f5e7d4b1940088bdec73c0b86fab.svg
Chenlu Ye
user
Chenlu123
67c12987505a88e4a185e0da
Lichang Chen
claimed_verified
2025-02-28T12:14:29.479Z
62323bb408bcea92917e42ee
https://cdn-avatars.huggingface.co/v1/production/uploads/62323bb408bcea92917e42ee/2vHxkv-oSROtLteOnqa8P.jpeg
Lichang Chen
user
Lichang-Chen
67c12987505a88e4a185e0db
Nan Jiang
admin_assigned
2025-02-28T12:23:02.992Z
64b8922ca1827cc8d04ae919
/avatars/0aaa83e3d09a82434e1d6af724aaa485.svg
Nan Jiang
user
nanjiang
67c12987505a88e4a185e0dc
Tong Zhang
Self-rewarding correction for mathematical reasoning
We study self-rewarding reasoning large language models (LLMs), which can
simultaneously generate step-by-step reasoning and evaluate the correctness of
their outputs during the inference time-without external feedback. This
integrated approach allows a single model to independently guide its reasoning
process, offering computational advantages for model deployment. We
particularly focus on the representative task of self-correction, where models
autonomously detect errors in their responses, revise outputs, and decide when
to terminate iterative refinement loops. To enable this, we propose a
two-staged algorithmic framework for constructing self-rewarding reasoning
models using only self-generated data. In the first stage, we employ sequential
rejection sampling to synthesize long chain-of-thought trajectories that
incorporate both self-rewarding and self-correction mechanisms. Fine-tuning
models on these curated data allows them to learn the patterns of
self-rewarding and self-correction. In the second stage, we further enhance the
models' ability to assess response accuracy and refine outputs through
reinforcement learning with rule-based signals. Experiments with Llama-3 and
Qwen-2.5 demonstrate that our approach surpasses intrinsic self-correction
capabilities and achieves performance comparable to systems that rely on
external reward models.
Self-rewarding reasoning large language models independently generate and correct their outputs during inference using a two-stage algorithmic framework, enhancing performance without external feedback.
self-rewarding reasoning large language models chain-of-thought trajectories sequential rejection sampling intrinsic self-correction reinforcement learning rule-based signals