The Illusion of Diminishing Returns: Measuring Long Horizon Execution in
  LLMs
Does continued scaling of large language models (LLMs) yield diminishing
returns? Real-world value often stems from the length of task an agent can
complete. We start this work by observing the simple but counterintuitive fact
that marginal gains in single-step accuracy can compound into exponential
improvements in the length of a task a model can successfully complete. Then,
we argue that failures of LLMs when simple tasks are made longer arise from
mistakes in execution, rather than an inability to reason. We propose isolating
execution capability, by explicitly providing the knowledge and plan needed to
solve a long-horizon task. We find that larger models can correctly execute
significantly more turns even when small models have 100\% single-turn
accuracy. We observe that the per-step accuracy of models degrades as the
number of steps increases. This is not just due to long-context limitations --
curiously, we observe a self-conditioning effect -- models become more likely
to make mistakes when the context contains their errors from prior turns.
Self-conditioning does not reduce by just scaling the model size. In contrast,
recent thinking models do not self-condition, and can also execute much longer
tasks in a single turn. We conclude by benchmarking frontier thinking models on
the length of task they can execute in a single turn. Overall, by focusing on
the ability to execute, we hope to reconcile debates on how LLMs can solve
complex reasoning problems yet fail at simple tasks when made longer, and
highlight the massive benefits of scaling model size and sequential test-time
compute for long-horizon tasks.
68c38a6afc1747b912403a3c
Akshit Sinha
claimed_verified
2025-09-12T16:08:01.121Z
651c184bdea81981d51158dd
/avatars/8afe17fcbd15d8ee767f24e4e8f34bbb.svg
Akshit Sinha
user
viciousa3gis
68c38a6afc1747b912403a3d
Arvindh Arun
claimed_verified
2025-09-12T16:07:57.509Z
62cd4dd7c5cc157be82f287a
/avatars/eb2e819dcdb67bafecbe0db3b1302c61.svg
Arvindh Arun
user
arvindh75
68c38a6afc1747b912403a3e
Shashwat Goel
68c38a6afc1747b912403a3f
Steffen Staab
68c38a6afc1747b912403a40
Jonas Geiping
The Illusion of Diminishing Returns: Measuring Long Horizon Execution in
  LLMs
Does continued scaling of large language models (LLMs) yield diminishing
returns? Real-world value often stems from the length of task an agent can
complete. We start this work by observing the simple but counterintuitive fact
that marginal gains in single-step accuracy can compound into exponential
improvements in the length of a task a model can successfully complete. Then,
we argue that failures of LLMs when simple tasks are made longer arise from
mistakes in execution, rather than an inability to reason. We propose isolating
execution capability, by explicitly providing the knowledge and plan needed to
solve a long-horizon task. We find that larger models can correctly execute
significantly more turns even when small models have 100\% single-turn
accuracy. We observe that the per-step accuracy of models degrades as the
number of steps increases. This is not just due to long-context limitations --
curiously, we observe a self-conditioning effect -- models become more likely
to make mistakes when the context contains their errors from prior turns.
Self-conditioning does not reduce by just scaling the model size. In contrast,
recent thinking models do not self-condition, and can also execute much longer
tasks in a single turn. We conclude by benchmarking frontier thinking models on
the length of task they can execute in a single turn. Overall, by focusing on
the ability to execute, we hope to reconcile debates on how LLMs can solve
complex reasoning problems yet fail at simple tasks when made longer, and
highlight the massive benefits of scaling model size and sequential test-time
compute for long-horizon tasks.
Scaling large language models improves their ability to execute longer tasks by isolating execution capability and mitigating self-conditioning effects, despite diminishing single-step accuracy.
large language models LLMs single-step accuracy long-horizon tasks execution capability self-conditioning thinking models sequential test-time compute