Training Language Models to Self-Correct via Reinforcement Learning
Self-correction is a highly desirable capability of large language models
(LLMs), yet it has consistently been found to be largely ineffective in modern
LLMs. Existing approaches for training self-correction either require multiple
models or rely on a more capable model or other forms of supervision. To this
end, we develop a multi-turn online reinforcement learning (RL) approach,
SCoRe, that significantly improves an LLM's self-correction ability using
entirely self-generated data. To build SCoRe, we first show that variants of
supervised fine-tuning (SFT) on offline model-generated correction traces are
insufficient for instilling self-correction behavior. In particular, we observe
that training via SFT either suffers from a distribution mismatch between the
training data and the model's own responses or implicitly prefers only a
certain mode of correction behavior that is often not effective at test time.
SCoRe addresses these challenges by training under the model's own distribution
of self-generated correction traces and using appropriate regularization to
steer the learning process into learning a self-correction strategy that is
effective at test time as opposed to simply fitting high-reward responses for a
given prompt. This regularization prescribes running a first phase of RL on a
base model to generate a policy initialization that is less susceptible to
collapse and then using a reward bonus to amplify self-correction during
training. When applied to Gemini 1.0 Pro and 1.5 Flash models, we find that
SCoRe achieves state-of-the-art self-correction performance, improving the base
models' self-correction by 15.6% and 9.1% respectively on the MATH and
HumanEval benchmarks.
66ed017df63ffc857d30af8c
Aviral Kumar
admin_assigned
2024-09-20T08:48:30.128Z
64879130cc0e54f272b5dbcc
/avatars/ead1211a56a4e5c14cb8f6b2ea59c0e6.svg
Aviral Kumar
user
aviralkumar
66ed017df63ffc857d30af8d
Vincent Zhuang
66ed017df63ffc857d30af8e
Rishabh Agarwal
admin_assigned
2024-09-20T08:48:50.526Z
65318c1022350ffeaf7726ef
/avatars/9359e63d38cb69d08da2de07ae933885.svg
Rishabh Agarwal
user
rishabhagarwal
66ed017df63ffc857d30af8f
Yi Su
66ed017df63ffc857d30af90
John D Co-Reyes
66ed017df63ffc857d30af91
Avi Singh
66ed017df63ffc857d30af92
Kate Baumli
66ed017df63ffc857d30af93
Shariq Iqbal
66ed017df63ffc857d30af94
Colton Bishop
66ed017df63ffc857d30af95
Rebecca Roelofs
66ed017df63ffc857d30af96
Lei M Zhang
66ed017df63ffc857d30af97
Kay McKinney
66ed017df63ffc857d30af98
Disha Shrivastava
admin_assigned
2024-09-20T08:49:52.218Z
6438284b819f3ab20d185a3b
https://cdn-avatars.huggingface.co/v1/production/uploads/6438284b819f3ab20d185a3b/b29SoH7_Bji6GfCszjqOE.jpeg
Disha Shrivastava
user
shrivasd
66ed017df63ffc857d30af99
Cosmin Paduraru
66ed017df63ffc857d30af9a
George Tucker
66ed017df63ffc857d30af9b
Doina Precup
66ed017df63ffc857d30af9c
Feryal Behbahani
claimed_verified
2024-09-20T07:45:36.990Z
64c197b7ec3c6181351b2c5d
/avatars/f81fe98f68720b95885d4968b0e7cc0f.svg
Feryal Behbahani
user
ferya
66ed017df63ffc857d30af9d
Aleksandra Faust
admin_assigned
2024-09-20T08:48:09.115Z
64a3b828225b2121f6b3beb3
https://cdn-avatars.huggingface.co/v1/production/uploads/64a3b828225b2121f6b3beb3/44AgAQENc7cJMOWcoK4g_.jpeg
Aleksandra Faust
user
sandraorion
Training Language Models to Self-Correct via Reinforcement Learning
Self-correction is a highly desirable capability of large language models
(LLMs), yet it has consistently been found to be largely ineffective in modern
LLMs. Existing approaches for training self-correction either require multiple
models or rely on a more capable model or other forms of supervision. To this
end, we develop a multi-turn online reinforcement learning (RL) approach,
SCoRe, that significantly improves an LLM's self-correction ability using
entirely self-generated data. To build SCoRe, we first show that variants of
supervised fine-tuning (SFT) on offline model-generated correction traces are
insufficient for instilling self-correction behavior. In particular, we observe
that training via SFT either suffers from a distribution mismatch between the
training data and the model's own responses or implicitly prefers only a
certain mode of correction behavior that is often not effective at test time.
SCoRe addresses these challenges by training under the model's own distribution
of self-generated correction traces and using appropriate regularization to
steer the learning process into learning a self-correction strategy that is
effective at test time as opposed to simply fitting high-reward responses for a
given prompt. This regularization prescribes running a first phase of RL on a
base model to generate a policy initialization that is less susceptible to
collapse and then using a reward bonus to amplify self-correction during
training. When applied to Gemini 1.0 Pro and 1.5 Flash models, we find that
SCoRe achieves state-of-the-art self-correction performance, improving the base
models' self-correction by 15.6% and 9.1% respectively on the MATH and
HumanEval benchmarks.
The multi-turn online reinforcement learning approach SCoRe improves LLM self-correction using self-generated data and regularization, achieving state-of-the-art performance.
reinforcement learning RL self-correction supervised fine-tuning SFT distribution mismatch reward bonus policy initialization Gemini 1.0 Pro Gemini 1.5 Flash MATH benchmark HumanEval benchmark