Quiet-STaR: Language Models Can Teach Themselves to Think Before
  Speaking
When writing and talking, people sometimes pause to think. Although
reasoning-focused works have often framed reasoning as a method of answering
questions or completing agentic tasks, reasoning is implicit in almost all
written text. For example, this applies to the steps not stated between the
lines of a proof or to the theory of mind underlying a conversation. In the
Self-Taught Reasoner (STaR, Zelikman et al. 2022), useful thinking is learned
by inferring rationales from few-shot examples in question-answering and
learning from those that lead to a correct answer. This is a highly constrained
setting -- ideally, a language model could instead learn to infer unstated
rationales in arbitrary text. We present Quiet-STaR, a generalization of STaR
in which LMs learn to generate rationales at each token to explain future text,
improving their predictions. We address key challenges, including 1) the
computational cost of generating continuations, 2) the fact that the LM does
not initially know how to generate or use internal thoughts, and 3) the need to
predict beyond individual next tokens. To resolve these, we propose a tokenwise
parallel sampling algorithm, using learnable tokens indicating a thought's
start and end, and an extended teacher-forcing technique. Encouragingly,
generated rationales disproportionately help model difficult-to-predict tokens
and improve the LM's ability to directly answer difficult questions. In
particular, after continued pretraining of an LM on a corpus of internet text
with Quiet-STaR, we find zero-shot improvements on GSM8K
(5.9%rightarrow10.9%) and CommonsenseQA (36.3%rightarrow47.2%) and
observe a perplexity improvement of difficult tokens in natural text.
Crucially, these improvements require no fine-tuning on these tasks. Quiet-STaR
marks a step towards LMs that can learn to reason in a more general and
scalable way.
65f3b1f154ecda1ecb3dde3c
Eric Zelikman
admin_assigned
2024-03-15T08:18:12.848Z
626aa3efe9326611a509a2fc
/avatars/4cea5cbd4ee918f1de7ccf191bfdf118.svg
Eric Zelikman
user
ezelikman
65f3b1f154ecda1ecb3dde3d
Georges Harik
admin_assigned
2024-03-15T08:18:22.832Z
6200d379a3f03cce621e1eaa
/avatars/d452b3bb0fe7f9299463ea551601eebf.svg
Georges Harik
user
gharik
65f3b1f154ecda1ecb3dde3e
Yijia Shao
claimed_verified
2024-09-05T07:24:55.599Z
6171514808b4ddb7392f2eb0
https://cdn-avatars.huggingface.co/v1/production/uploads/6171514808b4ddb7392f2eb0/6TjLoLeKwos0JoGcTueeI.jpeg
Yijia Shao
user
EchoShao8899
65f3b1f154ecda1ecb3dde3f
Varuna Jayasiri
admin_assigned
2024-03-15T08:18:37.809Z
62ceaf9654477562dcb4afda
https://cdn-avatars.huggingface.co/v1/production/uploads/62ceaf9654477562dcb4afda/cWLv6_qWAxY7_biXfjiGU.jpeg
Varuna Jayasiri
user
vpj
65f3b1f154ecda1ecb3dde40
Nick Haber
admin_assigned
2024-03-15T08:18:43.946Z
657d3b7d69cfe9b4cd0bfa04
/avatars/a97b0df7187d08ccca8a6a597cba7f67.svg
Nick Haber
user
nhaber
65f3b1f154ecda1ecb3dde41
Noah D. Goodman
claimed_verified
2024-11-15T09:49:42.038Z
67321274c1f20c742bcf7a8d
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/ltcQhre6eDRVzn6Vbbyhu.png
Noah D. Goodman
user
ngoodman
Quiet-STaR: Language Models Can Teach Themselves to Think Before
  Speaking
When writing and talking, people sometimes pause to think. Although
reasoning-focused works have often framed reasoning as a method of answering
questions or completing agentic tasks, reasoning is implicit in almost all
written text. For example, this applies to the steps not stated between the
lines of a proof or to the theory of mind underlying a conversation. In the
Self-Taught Reasoner (STaR, Zelikman et al. 2022), useful thinking is learned
by inferring rationales from few-shot examples in question-answering and
learning from those that lead to a correct answer. This is a highly constrained
setting -- ideally, a language model could instead learn to infer unstated
rationales in arbitrary text. We present Quiet-STaR, a generalization of STaR
in which LMs learn to generate rationales at each token to explain future text,
improving their predictions. We address key challenges, including 1) the
computational cost of generating continuations, 2) the fact that the LM does
not initially know how to generate or use internal thoughts, and 3) the need to
predict beyond individual next tokens. To resolve these, we propose a tokenwise
parallel sampling algorithm, using learnable tokens indicating a thought's
start and end, and an extended teacher-forcing technique. Encouragingly,
generated rationales disproportionately help model difficult-to-predict tokens
and improve the LM's ability to directly answer difficult questions. In
particular, after continued pretraining of an LM on a corpus of internet text
with Quiet-STaR, we find zero-shot improvements on GSM8K
(5.9%rightarrow10.9%) and CommonsenseQA (36.3%rightarrow47.2%) and
observe a perplexity improvement of difficult tokens in natural text.
Crucially, these improvements require no fine-tuning on these tasks. Quiet-STaR
marks a step towards LMs that can learn to reason in a more general and
scalable way.
Quiet-STaR is an extension of the Self-Taught Reasoner that teaches language models to infer and generate rationales for each token, improving their ability to predict difficult tokens and answer questions without task-specific fine-tuning.
Self-Taught Reasoner Quiet-STaR language models tokenwise parallel sampling internal thoughts extended teacher-forcing GSM8K CommonsenseQA perplexity reasoning