Understanding Tool-Integrated Reasoning
We study why Tool-Integrated Reasoning (TIR) makes Large Language Models
(LLMs) more capable. While LLMs integrated with tools like Python code
interpreters show great promise, a principled theory explaining why this
paradigm is effective has been missing. This work provides the first formal
proof that TIR fundamentally expands an LLM's capabilities. We demonstrate that
tools enable a strict expansion of the model's empirical and feasible support,
breaking the capability ceiling of pure-text models by unlocking
problem-solving strategies that are otherwise impossible or intractably
verbose. To guide model behavior without compromising training stability and
performance, we also introduce Advantage Shaping Policy Optimization (ASPO), a
novel algorithm that directly modifies the advantage function to guide the
policy behavior. We conduct comprehensive experiments on challenging
mathematical benchmarks, leveraging a Python interpreter as the external tool.
Our results show that the TIR model decisively outperforms its pure-text
counterpart on the pass@k metric. Crucially, this advantage is not confined to
computationally-intensive problems but extends to those requiring significant
abstract insight. We further identify the emergent cognitive patterns that
illustrate how models learn to think with tools. Finally, we report improved
tool usage behavior with early code invocation and much more interactive turns
with ASPO. Overall, our work provides the first principled explanation for
TIR's success, shifting the focus from the mere fact that tools work to why and
how they enable more powerful reasoning.
68ae5c89364411bea07df6ba
Heng Lin
claimed_verified
2025-08-27T12:45:48.349Z
68ac517e6843e1930afeb683
/avatars/ced835ec3b1a1ca2c88ef641a32278db.svg
Heng Lin
user
Heng1999
68ae5c89364411bea07df6bb
Zhongwen Xu
claimed_verified
2025-08-27T07:11:06.169Z
672db76fa34d64e774fc42c9
/avatars/be529f4bfcc40514697facbe8e874735.svg
Zhongwen Xu
user
zhongwenxu
Understanding Tool-Integrated Reasoning
We study why Tool-Integrated Reasoning (TIR) makes Large Language Models
(LLMs) more capable. While LLMs integrated with tools like Python code
interpreters show great promise, a principled theory explaining why this
paradigm is effective has been missing. This work provides the first formal
proof that TIR fundamentally expands an LLM's capabilities. We demonstrate that
tools enable a strict expansion of the model's empirical and feasible support,
breaking the capability ceiling of pure-text models by unlocking
problem-solving strategies that are otherwise impossible or intractably
verbose. To guide model behavior without compromising training stability and
performance, we also introduce Advantage Shaping Policy Optimization (ASPO), a
novel algorithm that directly modifies the advantage function to guide the
policy behavior. We conduct comprehensive experiments on challenging
mathematical benchmarks, leveraging a Python interpreter as the external tool.
Our results show that the TIR model decisively outperforms its pure-text
counterpart on the pass@k metric. Crucially, this advantage is not confined to
computationally-intensive problems but extends to those requiring significant
abstract insight. We further identify the emergent cognitive patterns that
illustrate how models learn to think with tools. Finally, we report improved
tool usage behavior with early code invocation and much more interactive turns
with ASPO. Overall, our work provides the first principled explanation for
TIR's success, shifting the focus from the mere fact that tools work to why and
how they enable more powerful reasoning.
Tool-Integrated Reasoning (TIR) enhances Large Language Models (LLMs) by expanding their problem-solving capabilities through the use of external tools, and Advantage Shaping Policy Optimization (ASPO) improves model behavior and tool usage.
Tool-Integrated Reasoning Large Language Models LLMs Advantage Shaping Policy Optimization ASPO empirical support feasible support capability ceiling problem-solving strategies pass@k metric cognitive patterns code invocation interactive turns