ToolRL: Reward is All Tool Learning Needs
Current Large Language Models (LLMs) often undergo supervised fine-tuning
(SFT) to acquire tool use capabilities. However, SFT struggles to generalize to
unfamiliar or complex tool use scenarios. Recent advancements in reinforcement
learning (RL), particularly with R1-like models, have demonstrated promising
reasoning and generalization abilities. Yet, reward design for tool use
presents unique challenges: multiple tools may be invoked with diverse
parameters, and coarse-grained reward signals, such as answer matching, fail to
offer the finegrained feedback required for effective learning. In this work,
we present the first comprehensive study on reward design for tool selection
and application tasks within the RL paradigm. We systematically explore a wide
range of reward strategies, analyzing their types, scales, granularity, and
temporal dynamics. Building on these insights, we propose a principled reward
design tailored for tool use tasks and apply it to train LLMs using Group
Relative Policy Optimization (GRPO). Empirical evaluations across diverse
benchmarks demonstrate that our approach yields robust, scalable, and stable
training, achieving a 17% improvement over base models and a 15% gain over SFT
models. These results highlight the critical role of thoughtful reward design
in enhancing the tool use capabilities and generalization performance of LLMs.
All the codes are released to facilitate future research.
6806fe1bec9fb3764b875ea0
Cheng Qian
claimed_verified
2025-04-23T08:37:34.019Z
665e121c6007027038fd4005
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/sIVBJAGM-Kneq9KMf8aXb.png
Cheng Qian
user
chengq9
6806fe1bec9fb3764b875ea1
Emre Can Acikgoz
claimed_verified
2025-04-22T09:50:57.521Z
63888d3fd68e37abd599f428
https://cdn-avatars.huggingface.co/v1/production/uploads/63888d3fd68e37abd599f428/YaNyxG_oM6IgrHTkFZ6Eq.jpeg
emre can
user
emrecanacikgoz
6806fe1bec9fb3764b875ea2
Qi He
6806fe1bec9fb3764b875ea3
Hongru Wang
admin_assigned
2025-04-22T10:52:19.954Z
65f906e5c3dbdcae83ff7aac
https://cdn-avatars.huggingface.co/v1/production/uploads/65f906e5c3dbdcae83ff7aac/mdjiVkLDJgJcGLwv0rMe4.jpeg
Hongru Wang
user
Merlin-Hongru
6806fe1bec9fb3764b875ea4
Xiusi Chen
admin_assigned
2025-04-22T10:51:58.709Z
6270ff726417aed8a7340c8b
/avatars/3f14913c55cc4fc78678ac43fb603e80.svg
Xiusi Chen
user
XtremSup
6806fe1bec9fb3764b875ea5
Dilek Hakkani-Tür
claimed_verified
2025-06-16T15:51:11.174Z
6747519282137543033451d9
/avatars/8bd763f19d8e9f97f0f6817435b062dc.svg
Dilek Hakkani-Tur
user
dilekht
6806fe1bec9fb3764b875ea6
Gokhan Tur
admin_assigned
2025-04-22T10:51:42.581Z
640f687c06c3b5ca88412985
/avatars/6c1e538b78f52ce45ff124b0a5ec3369.svg
gokhan tur
user
Gokhantur
6806fe1bec9fb3764b875ea7
Heng Ji
ToolRL: Reward is All Tool Learning Needs
Current Large Language Models (LLMs) often undergo supervised fine-tuning
(SFT) to acquire tool use capabilities. However, SFT struggles to generalize to
unfamiliar or complex tool use scenarios. Recent advancements in reinforcement
learning (RL), particularly with R1-like models, have demonstrated promising
reasoning and generalization abilities. Yet, reward design for tool use
presents unique challenges: multiple tools may be invoked with diverse
parameters, and coarse-grained reward signals, such as answer matching, fail to
offer the finegrained feedback required for effective learning. In this work,
we present the first comprehensive study on reward design for tool selection
and application tasks within the RL paradigm. We systematically explore a wide
range of reward strategies, analyzing their types, scales, granularity, and
temporal dynamics. Building on these insights, we propose a principled reward
design tailored for tool use tasks and apply it to train LLMs using Group
Relative Policy Optimization (GRPO). Empirical evaluations across diverse
benchmarks demonstrate that our approach yields robust, scalable, and stable
training, achieving a 17% improvement over base models and a 15% gain over SFT
models. These results highlight the critical role of thoughtful reward design
in enhancing the tool use capabilities and generalization performance of LLMs.
All the codes are released to facilitate future research.
A comprehensive study on reward design for tool use within reinforcement learning improves LLMs' tool use capabilities and generalization performance over supervised fine-tuning.
supervised fine-tuning reinforcement learning R1-like models reward design tool selection application tasks reward strategies Group Relative Policy Optimization (GRPO)