ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world
  APIs
Despite the advancements of open-source large language models (LLMs) and
their variants, e.g., LLaMA and Vicuna, they remain significantly limited in
performing higher-level tasks, such as following human instructions to use
external tools (APIs). This is because current instruction tuning largely
focuses on basic language tasks instead of the tool-use domain. This is in
contrast to state-of-the-art (SOTA) LLMs, e.g., ChatGPT, which have
demonstrated excellent tool-use capabilities but are unfortunately closed
source. To facilitate tool-use capabilities within open-source LLMs, we
introduce ToolLLM, a general tool-use framework of data construction, model
training and evaluation. We first present ToolBench, an instruction-tuning
dataset for tool use, which is created automatically using ChatGPT.
Specifically, we collect 16,464 real-world RESTful APIs spanning 49 categories
from RapidAPI Hub, then prompt ChatGPT to generate diverse human instructions
involving these APIs, covering both single-tool and multi-tool scenarios.
Finally, we use ChatGPT to search for a valid solution path (chain of API
calls) for each instruction. To make the searching process more efficient, we
develop a novel depth-first search-based decision tree (DFSDT), enabling LLMs
to evaluate multiple reasoning traces and expand the search space. We show that
DFSDT significantly enhances the planning and reasoning capabilities of LLMs.
For efficient tool-use assessment, we develop an automatic evaluator: ToolEval.
We fine-tune LLaMA on ToolBench and obtain ToolLLaMA. Our ToolEval reveals that
ToolLLaMA demonstrates a remarkable ability to execute complex instructions and
generalize to unseen APIs, and exhibits comparable performance to ChatGPT. To
make the pipeline more practical, we devise a neural API retriever to recommend
appropriate APIs for each instruction, negating the need for manual API
selection.
64c85d053d5a0dfed5b65fc9
Yujia Qin
extracted_confirmed
2023-08-01T01:18:13.843Z
643f37cce9d063936912048b
/avatars/25822ea5676a79b2e1ddf08d5fc2226c.svg
Yujia Qin
user
YujiaHi
64c85d053d5a0dfed5b65fca
Shihao Liang
claimed_verified
2023-08-01T07:44:43.257Z
6241d6b4bdf59d13c0a8dd86
/avatars/f729f0bdcfbd971babfa760d8de8403c.svg
Shihao Liang
user
pooruss
64c85d053d5a0dfed5b65fcb
Yining Ye
claimed_verified
2023-08-02T07:39:26.901Z
636a0803fbc31ee68fed16bd
/avatars/331073d0fee901b2b9d93287fabe64fc.svg
yining ye
user
yeyn2001
64c85d053d5a0dfed5b65fcc
Kunlun Zhu
admin_assigned
2023-08-01T07:46:35.841Z
64c090a9f613170e7be93d2f
/avatars/ccbdf444e1f2386d2281e8e42059ebb0.svg
KunlunZhu
user
KunlunZhu
64c85d053d5a0dfed5b65fcd
Lan Yan
64c85d053d5a0dfed5b65fce
Yaxi Lu
64c85d053d5a0dfed5b65fcf
Yankai Lin
64c85d053d5a0dfed5b65fd0
Xin Cong
64c85d053d5a0dfed5b65fd1
Xiangru Tang
claimed_verified
2024-02-07T11:17:54.591Z
63357c608adfa81faf2ac180
/avatars/ae0314c644f882251baf59b9134fd36f.svg
Xiangru Tang
user
RTT1
64c85d053d5a0dfed5b65fd2
Bill Qian
admin_assigned
2023-08-01T07:48:25.080Z
641fa2e0863b87326f4a7fd5
/avatars/7b29c20f942d83c08f63f04a85c15c13.svg
Bill Qian
user
lilbillbiscuit
64c85d053d5a0dfed5b65fd3
Sihan Zhao
64c85d053d5a0dfed5b65fd4
Runchu Tian
64c85d053d5a0dfed5b65fd5
Ruobing Xie
claimed_verified
2024-11-05T09:02:01.538Z
6622443b9b0614a760dd8123
/avatars/acb6c1c9c429af1112530dcf76a8e420.svg
Ruobing Xie
user
Ruobing-Xie
64c85d053d5a0dfed5b65fd6
Jie Zhou
64c85d053d5a0dfed5b65fd7
Mark Gerstein
64c85d053d5a0dfed5b65fd8
Dahai Li
64c85d053d5a0dfed5b65fd9
Zhiyuan Liu
claimed_verified
2025-06-02T08:01:59.991Z
6472f1f7485a7c8e1bcf507b
/avatars/79dca137583875191ea883b193b630b7.svg
Zhiyuan Liu
user
zibuyu9
64c85d053d5a0dfed5b65fda
Maosong Sun
ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world
  APIs
Despite the advancements of open-source large language models (LLMs) and
their variants, e.g., LLaMA and Vicuna, they remain significantly limited in
performing higher-level tasks, such as following human instructions to use
external tools (APIs). This is because current instruction tuning largely
focuses on basic language tasks instead of the tool-use domain. This is in
contrast to state-of-the-art (SOTA) LLMs, e.g., ChatGPT, which have
demonstrated excellent tool-use capabilities but are unfortunately closed
source. To facilitate tool-use capabilities within open-source LLMs, we
introduce ToolLLM, a general tool-use framework of data construction, model
training and evaluation. We first present ToolBench, an instruction-tuning
dataset for tool use, which is created automatically using ChatGPT.
Specifically, we collect 16,464 real-world RESTful APIs spanning 49 categories
from RapidAPI Hub, then prompt ChatGPT to generate diverse human instructions
involving these APIs, covering both single-tool and multi-tool scenarios.
Finally, we use ChatGPT to search for a valid solution path (chain of API
calls) for each instruction. To make the searching process more efficient, we
develop a novel depth-first search-based decision tree (DFSDT), enabling LLMs
to evaluate multiple reasoning traces and expand the search space. We show that
DFSDT significantly enhances the planning and reasoning capabilities of LLMs.
For efficient tool-use assessment, we develop an automatic evaluator: ToolEval.
We fine-tune LLaMA on ToolBench and obtain ToolLLaMA. Our ToolEval reveals that
ToolLLaMA demonstrates a remarkable ability to execute complex instructions and
generalize to unseen APIs, and exhibits comparable performance to ChatGPT. To
make the pipeline more practical, we devise a neural API retriever to recommend
appropriate APIs for each instruction, negating the need for manual API
selection.
ToolLLM is a framework that enhances open-source LLMs with tool-use capabilities through a comprehensive dataset, efficient decision tree, automatic evaluation, and neural API retrieval.
instruction tuning tool-use framework ToolBench RESTful APIs depth-first search-based decision tree ToolEval fine-tuning API retriever