Toward Effective Tool-Integrated Reasoning via Self-Evolved Preference
  Learning
Tool-Integrated Reasoning (TIR) enables large language models (LLMs) to
improve their internal reasoning ability by integrating external tools.
However, models employing TIR often display suboptimal behaviors, such as
insufficient or excessive tool usage and overthinking after tool calls. The
challenge of incentivizing LLMs to perform TIR efficiently and accurately,
while stabilizing the reasoning process, remains an open question. In this
paper, we start by exploring the impact of tool calls on model reasoning from
the perspective of information entropy. Our findings indicate that tool call
results lead to a distinct change in the information entropy of subsequent
reasoning, with the overall entropy of the reasoning chain varying based on the
number of tool calls. Building on these insights, we propose Tool-Light, a
framework designed to encourage LLMs to perform TIR efficiently and accurately.
Our framework includes dataset construction and multi-stage fine-tuning. For
dataset construction, we employ continuous self-evolved sampling using the
fine-tuned model, integrating both vanilla sampling and entropy-guided
sampling. Besides, we establish strict criteria for selecting positive-negative
pairs during sampling. The training process involves a two-stage approach,
comprising Supervised Fine-Tuning (SFT) and Self-Evolved Direct Preference
Optimization (DPO). Experimental results on 10 datasets demonstrate the
effectiveness of Tool-Light, significantly improving the model's efficiency in
executing TIR tasks.
68db3809d2bf1f4b15ec728b
Yifei Chen
admin_assigned
2025-10-14T16:58:12.870Z
6621ec2524eb2673fe0790fc
https://cdn-avatars.huggingface.co/v1/production/uploads/6621ec2524eb2673fe0790fc/cooTXi12eRWFiSSIj_nA-.jpeg
Ania Forge
user
zhangboguodong
68db3809d2bf1f4b15ec728c
Guanting Dong
claimed_verified
2025-10-17T04:14:44.377Z
61cd4b833dd34ba1985e0753
https://cdn-avatars.huggingface.co/v1/production/uploads/61cd4b833dd34ba1985e0753/BfHfrwotoMESpXZOHiIe4.png
KABI
user
dongguanting
68db3809d2bf1f4b15ec728d
Zhicheng Dou
Toward Effective Tool-Integrated Reasoning via Self-Evolved Preference
  Learning
Tool-Integrated Reasoning (TIR) enables large language models (LLMs) to
improve their internal reasoning ability by integrating external tools.
However, models employing TIR often display suboptimal behaviors, such as
insufficient or excessive tool usage and overthinking after tool calls. The
challenge of incentivizing LLMs to perform TIR efficiently and accurately,
while stabilizing the reasoning process, remains an open question. In this
paper, we start by exploring the impact of tool calls on model reasoning from
the perspective of information entropy. Our findings indicate that tool call
results lead to a distinct change in the information entropy of subsequent
reasoning, with the overall entropy of the reasoning chain varying based on the
number of tool calls. Building on these insights, we propose Tool-Light, a
framework designed to encourage LLMs to perform TIR efficiently and accurately.
Our framework includes dataset construction and multi-stage fine-tuning. For
dataset construction, we employ continuous self-evolved sampling using the
fine-tuned model, integrating both vanilla sampling and entropy-guided
sampling. Besides, we establish strict criteria for selecting positive-negative
pairs during sampling. The training process involves a two-stage approach,
comprising Supervised Fine-Tuning (SFT) and Self-Evolved Direct Preference
Optimization (DPO). Experimental results on 10 datasets demonstrate the
effectiveness of Tool-Light, significantly improving the model's efficiency in
executing TIR tasks.
Tool-Light framework improves large language models' tool-integrated reasoning efficiency and accuracy by leveraging information entropy and a two-stage fine-tuning process.
Tool-Integrated Reasoning large language models information entropy Tool-Light dataset construction continuous self-evolved sampling entropy-guided sampling positive-negative pairs Supervised Fine-Tuning Self-Evolved Direct Preference Optimization