OTC: Optimal Tool Calls via Reinforcement Learning
Tool-integrated reasoning (TIR) augments large language models (LLMs) with
the ability to invoke external tools, such as search engines and code
interpreters, to solve tasks beyond the capabilities of language-only
reasoning. While reinforcement learning (RL) has shown promise in improving TIR
by optimizing final answer correctness, existing approaches often overlook the
efficiency and cost associated with tool usage. This can lead to suboptimal
behavior, including excessive tool calls that increase computational and
financial overhead, or insufficient tool use that compromises answer quality.
In this work, we propose Optimal Tool Call-controlled Policy Optimization
(OTC-PO), a simple yet effective RL-based framework that encourages models to
produce accurate answers with minimal tool calls. Our method introduces a
tool-integrated reward that jointly considers correctness and tool efficiency,
promoting high tool productivity. We instantiate this framework within both
Proximal Policy Optimization (PPO) and Group Relative Preference Optimization
(GRPO), resulting in OTC-PPO and OTC-GRPO. Experiments with Qwen-2.5 and
Qwen-Math across multiple QA benchmarks show that our approach reduces tool
calls by up to 73.1\% and improves tool productivity by up to 229.4\%, while
maintaining comparable answer accuracy. To the best of our knowledge, this is
the first RL-based framework that explicitly optimizes tool-use efficiency in
TIR.
68070933142b618d3357d2bc
Hongru Wang
claimed_verified
2025-04-28T12:50:29.698Z
65f906e5c3dbdcae83ff7aac
https://cdn-avatars.huggingface.co/v1/production/uploads/65f906e5c3dbdcae83ff7aac/mdjiVkLDJgJcGLwv0rMe4.jpeg
Hongru Wang
user
Merlin-Hongru
68070933142b618d3357d2bd
Cheng Qian
claimed_verified
2025-04-23T08:37:30.075Z
665e121c6007027038fd4005
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/sIVBJAGM-Kneq9KMf8aXb.png
Cheng Qian
user
chengq9
68070933142b618d3357d2be
Wanjun Zhong
68070933142b618d3357d2bf
Xiusi Chen
68070933142b618d3357d2c0
Jiahao Qiu
claimed_verified
2025-05-28T19:14:57.203Z
65271df5abd7795aaa1fd86e
/avatars/99e1aabaefcf5e7f438817375cd1ddef.svg
Jiahao Qiu
user
jiahaoq
68070933142b618d3357d2c1
Shijue Huang
68070933142b618d3357d2c2
Bowen Jin
68070933142b618d3357d2c3
Mengdi Wang
68070933142b618d3357d2c4
Kam-Fai Wong
68070933142b618d3357d2c5
Heng Ji
OTC: Optimal Tool Calls via Reinforcement Learning
Tool-integrated reasoning (TIR) augments large language models (LLMs) with
the ability to invoke external tools, such as search engines and code
interpreters, to solve tasks beyond the capabilities of language-only
reasoning. While reinforcement learning (RL) has shown promise in improving TIR
by optimizing final answer correctness, existing approaches often overlook the
efficiency and cost associated with tool usage. This can lead to suboptimal
behavior, including excessive tool calls that increase computational and
financial overhead, or insufficient tool use that compromises answer quality.
In this work, we propose Optimal Tool Call-controlled Policy Optimization
(OTC-PO), a simple yet effective RL-based framework that encourages models to
produce accurate answers with minimal tool calls. Our method introduces a
tool-integrated reward that jointly considers correctness and tool efficiency,
promoting high tool productivity. We instantiate this framework within both
Proximal Policy Optimization (PPO) and Group Relative Preference Optimization
(GRPO), resulting in OTC-PPO and OTC-GRPO. Experiments with Qwen-2.5 and
Qwen-Math across multiple QA benchmarks show that our approach reduces tool
calls by up to 73.1\% and improves tool productivity by up to 229.4\%, while
maintaining comparable answer accuracy. To the best of our knowledge, this is
the first RL-based framework that explicitly optimizes tool-use efficiency in
TIR.
A novel RL framework optimizes tool usage in language models for enhanced efficiency and cost-effectiveness without sacrificing accuracy.
tool-integrated reasoning LLMs reinforcement learning RL Proximal Policy Optimization PPO Group Relative Preference Optimization GRPO tool-integrated reward tool productivity QA benchmarks