Rethinking Thinking Tokens: LLMs as Improvement Operators
Reasoning training incentivizes LLMs to produce long chains of thought (long
CoT), which among other things, allows them to explore solution strategies with
self-checking. This results in higher accuracy, but inflates context length,
token/compute cost, and answer latency. We ask: Can current models leverage
their metacognition to provide other combinations on this Pareto frontier,
e.g., better accuracy with lower context length and/or latency? Abstractly, we
view the model as an improvement operator on its own "thoughts" with a
continuum of possible strategies. We identify an interesting inference family
Parallel-Distill-Refine (PDR), which performs the following: (i) generate
diverse drafts in parallel; (ii) distill them into a bounded, textual
workspace; and (iii) refine conditioned on this workspace, producing an output
that seeds the next round. Importantly, context length (hence compute cost) is
controllable via degree of parallelism, and is no longer conflated with the
total number of generated tokens. We report PDR instantiations of current
models that give better accuracy than long CoT while incurring lower latency.
Setting degree of parallelism to 1 yields an interesting subcase, Sequential
Refinement (SR) (iteratively improve a single candidate answer) which provides
performance superior to long CoT. Success of such model orchestrations raises
the question whether further training could shift the Pareto frontier. To this
end, we train an 8B thinking model with Reinforcement Learning (RL) to make it
consistent with PDR as the inference method. On math tasks with verifiable
answers, iterative pipelines surpass single-pass baselines at matched
sequential budgets, with PDR delivering the largest gains (e.g., +11% on AIME
2024 and +9% on AIME 2025).
68e043c473e20ab577841b8b
Lovish Madaan
68e043c473e20ab577841b8c
Aniket Didolkar
68e043c473e20ab577841b8d
Suchin Gururangan
68e043c473e20ab577841b8e
John Quan
68e043c473e20ab577841b8f
Ruan Silva
68e043c473e20ab577841b90
Ruslan Salakhutdinov
68e043c473e20ab577841b91
Manzil Zaheer
68e043c473e20ab577841b92
Sanjeev Arora
68e043c473e20ab577841b93
Anirudh Goyal
Rethinking Thinking Tokens: LLMs as Improvement Operators
Reasoning training incentivizes LLMs to produce long chains of thought (long
CoT), which among other things, allows them to explore solution strategies with
self-checking. This results in higher accuracy, but inflates context length,
token/compute cost, and answer latency. We ask: Can current models leverage
their metacognition to provide other combinations on this Pareto frontier,
e.g., better accuracy with lower context length and/or latency? Abstractly, we
view the model as an improvement operator on its own "thoughts" with a
continuum of possible strategies. We identify an interesting inference family
Parallel-Distill-Refine (PDR), which performs the following: (i) generate
diverse drafts in parallel; (ii) distill them into a bounded, textual
workspace; and (iii) refine conditioned on this workspace, producing an output
that seeds the next round. Importantly, context length (hence compute cost) is
controllable via degree of parallelism, and is no longer conflated with the
total number of generated tokens. We report PDR instantiations of current
models that give better accuracy than long CoT while incurring lower latency.
Setting degree of parallelism to 1 yields an interesting subcase, Sequential
Refinement (SR) (iteratively improve a single candidate answer) which provides
performance superior to long CoT. Success of such model orchestrations raises
the question whether further training could shift the Pareto frontier. To this
end, we train an 8B thinking model with Reinforcement Learning (RL) to make it
consistent with PDR as the inference method. On math tasks with verifiable
answers, iterative pipelines surpass single-pass baselines at matched
sequential budgets, with PDR delivering the largest gains (e.g., +11% on AIME
2024 and +9% on AIME 2025).
Parallel-Distill-Refine (PDR) and Sequential Refinement (SR) improve the performance of LLMs by optimizing accuracy and latency through metacognitive strategies, with PDR showing significant gains on math tasks.
LLMs long chains of thought long CoT self-checking context length token/compute cost answer latency metacognition improvement operator Parallel-Distill-Refine PDR bounded textual workspace refine degree of parallelism Sequential Refinement SR Reinforcement Learning RL AIME 2024 AIME 2025