ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models
  in Multi-Hop Tool Use
Effective evaluation of multi-hop tool use is critical for analyzing the
understanding, reasoning, and function-calling capabilities of large language
models (LLMs). However, progress has been hindered by a lack of reliable
evaluation datasets. To address this, we present ToolHop, a dataset comprising
995 user queries and 3,912 associated tools, specifically designed for rigorous
evaluation of multi-hop tool use. ToolHop ensures diverse queries, meaningful
interdependencies, locally executable tools, detailed feedback, and verifiable
answers through a novel query-driven data construction approach that includes
tool creation, document refinement, and code generation. We evaluate 14 LLMs
across five model families (i.e., LLaMA3.1, Qwen2.5, Gemini1.5, Claude3.5, and
GPT), uncovering significant challenges in handling multi-hop tool-use
scenarios. The leading model, GPT-4o, achieves an accuracy of 49.04%,
underscoring substantial room for improvement. Further analysis reveals
variations in tool-use strategies for various families, offering actionable
insights to guide the development of more effective approaches. Code and data
can be found in https://huggingface.co/bytedance-research/ToolHop.
677cab009909f23ac4dc1d0e
Junjie Ye
claimed_verified
2025-01-07T13:02:00.881Z
66384be673c2c55f2ded89fa
/avatars/1d8721074f0f51fab405f81474f2035f.svg
Junjie Ye
user
Junjie-Ye
677cab009909f23ac4dc1d0f
Zhengyin Du
677cab009909f23ac4dc1d10
Xuesong Yao
claimed_verified
2025-04-03T08:29:49.648Z
656d8ec3fa91c81609822b96
/avatars/56f6eed2cdd4706db45ea6b20059265d.svg
Yao Xuesong
user
NathanYao
677cab009909f23ac4dc1d11
Weijian Lin
claimed_verified
2025-01-07T08:42:22.956Z
6511f4fdc915724a65380f78
/avatars/b7cf645e0a035e3a3683ee1fd01380ed.svg
Weijian Lin
user
maverick1994
677cab009909f23ac4dc1d12
Yufei Xu
claimed_verified
2025-04-02T08:52:15.015Z
642957104e073875f6a5ddd0
/avatars/32b706d35c5ff52932c5029b94caa7b9.svg
Yufei Xu
user
yfxu
677cab009909f23ac4dc1d13
Zehui Chen
677cab009909f23ac4dc1d14
Zaiyuan Wang
677cab009909f23ac4dc1d15
Sining Zhu
677cab009909f23ac4dc1d16
Zhiheng Xi
admin_assigned
2025-01-07T13:26:27.253Z
653a6e5cae155b92bae77b74
https://cdn-avatars.huggingface.co/v1/production/uploads/653a6e5cae155b92bae77b74/TA5FWKAUsB249ux4MzD_R.jpeg
Zhiheng Xi
user
WooooDyy
677cab009909f23ac4dc1d17
Siyu Yuan
admin_assigned
2025-01-07T13:26:42.842Z
62d62b333bf5e059f7d2b286
https://cdn-avatars.huggingface.co/v1/production/uploads/1668513815771-62d62b333bf5e059f7d2b286.jpeg
Siyu Yuan
user
siyuyuan
677cab009909f23ac4dc1d18
Tao Gui
admin_assigned
2025-01-07T13:26:50.647Z
65b71b9e8bb894c96cc7707b
/avatars/78e2a833054e89bcc34ddc6e789677d5.svg
Tao Gui
user
guixiaotao
677cab009909f23ac4dc1d19
Qi Zhang
677cab009909f23ac4dc1d1a
Xuanjing Huang
677cab009909f23ac4dc1d1b
Jiechao Chen
ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models
  in Multi-Hop Tool Use
Effective evaluation of multi-hop tool use is critical for analyzing the
understanding, reasoning, and function-calling capabilities of large language
models (LLMs). However, progress has been hindered by a lack of reliable
evaluation datasets. To address this, we present ToolHop, a dataset comprising
995 user queries and 3,912 associated tools, specifically designed for rigorous
evaluation of multi-hop tool use. ToolHop ensures diverse queries, meaningful
interdependencies, locally executable tools, detailed feedback, and verifiable
answers through a novel query-driven data construction approach that includes
tool creation, document refinement, and code generation. We evaluate 14 LLMs
across five model families (i.e., LLaMA3.1, Qwen2.5, Gemini1.5, Claude3.5, and
GPT), uncovering significant challenges in handling multi-hop tool-use
scenarios. The leading model, GPT-4o, achieves an accuracy of 49.04%,
underscoring substantial room for improvement. Further analysis reveals
variations in tool-use strategies for various families, offering actionable
insights to guide the development of more effective approaches. Code and data
can be found in https://huggingface.co/bytedance-research/ToolHop.
ToolHop, a new dataset for evaluating multi-hop tool use in large language models, highlights significant challenges and variations in tool-use strategies across different model families.
multi-hop tool use large language models (LLMs) ToolHop user queries tool creation document refinement code generation accuracy GPT-4o model families LLaMA3.1 Qwen2.5 Gemini1.5 Claude3.5 GPT