Physics of Language Models: Part 2.2, How to Learn From Mistakes on
  Grade-School Math Problems
Language models have demonstrated remarkable performance in solving reasoning
tasks; however, even the strongest models still occasionally make reasoning
mistakes. Recently, there has been active research aimed at improving reasoning
accuracy, particularly by using pretrained language models to "self-correct"
their mistakes via multi-round prompting. In this paper, we follow this line of
work but focus on understanding the usefulness of incorporating
"error-correction" data directly into the pretraining stage. This data consists
of erroneous solution steps immediately followed by their corrections. Using a
synthetic math dataset, we show promising results: this type of pretrain data
can help language models achieve higher reasoning accuracy directly (i.e.,
through simple auto-regression, without multi-round prompting) compared to
pretraining on the same amount of error-free data. We also delve into many
details, such as (1) how this approach differs from beam search, (2) how such
data can be prepared, (3) whether masking is needed on the erroneous tokens,
(4) the amount of error required, (5) whether such data can be deferred to the
fine-tuning stage, and many others.
66d23dd1960d799f1a69527b
Tian Ye
66d23dd1960d799f1a69527c
Zicheng Xu
claimed_verified
2024-09-02T07:17:41.699Z
66c7fc61d5a538bbb9e562b7
/avatars/feac2e2273be88a7a44f09adb4f373e2.svg
Zicheng Xu
user
xu2333
66d23dd1960d799f1a69527d
Yuanzhi Li
66d23dd1960d799f1a69527e
Zeyuan Allen-Zhu
claimed_verified
2024-09-02T07:17:44.222Z
6316d14c0db6c679cf77fe1f
/avatars/b33dcadc0ff8941da6aa850284be3207.svg
Zeyuan Allen-Zhu
user
zhuzeyuan
Physics of Language Models: Part 2.2, How to Learn From Mistakes on
  Grade-School Math Problems
Language models have demonstrated remarkable performance in solving reasoning
tasks; however, even the strongest models still occasionally make reasoning
mistakes. Recently, there has been active research aimed at improving reasoning
accuracy, particularly by using pretrained language models to "self-correct"
their mistakes via multi-round prompting. In this paper, we follow this line of
work but focus on understanding the usefulness of incorporating
"error-correction" data directly into the pretraining stage. This data consists
of erroneous solution steps immediately followed by their corrections. Using a
synthetic math dataset, we show promising results: this type of pretrain data
can help language models achieve higher reasoning accuracy directly (i.e.,
through simple auto-regression, without multi-round prompting) compared to
pretraining on the same amount of error-free data. We also delve into many
details, such as (1) how this approach differs from beam search, (2) how such
data can be prepared, (3) whether masking is needed on the erroneous tokens,
(4) the amount of error required, (5) whether such data can be deferred to the
fine-tuning stage, and many others.
Incorporating error-correction data into the pretraining of language models improves reasoning accuracy without multi-round prompting.
language models reasoning accuracy self-correction error-correction data synthetic math dataset auto-regression beam search masked tokens fine-tuning stage