MTU-Bench: A Multi-granularity Tool-Use Benchmark for Large Language
  Models
Large Language Models (LLMs) have displayed massive improvements in reasoning
and decision-making skills and can hold natural conversations with users.
Recently, many tool-use benchmark datasets have been proposed. However,
existing datasets have the following limitations: (1). Insufficient evaluation
scenarios (e.g., only cover limited tool-use scenes). (2). Extensive evaluation
costs (e.g., GPT API costs). To address these limitations, in this work, we
propose a multi-granularity tool-use benchmark for large language models called
MTU-Bench. For the "multi-granularity" property, our MTU-Bench covers five tool
usage scenes (i.e., single-turn and single-tool, single-turn and multiple-tool,
multiple-turn and single-tool, multiple-turn and multiple-tool, and
out-of-distribution tasks). Besides, all evaluation metrics of our MTU-Bench
are based on the prediction results and the ground truth without using any GPT
or human evaluation metrics. Moreover, our MTU-Bench is collected by
transforming existing high-quality datasets to simulate real-world tool usage
scenarios, and we also propose an instruction dataset called MTU-Instruct data
to enhance the tool-use abilities of existing LLMs. Comprehensive experimental
results demonstrate the effectiveness of our MTU-Bench. Code and data will be
released at https: //github.com/MTU-Bench-Team/MTU-Bench.git.
670f270f3240681ca35b0f9e
Pei Wang
670f270f3240681ca35b0f9f
Yanan Wu
670f270f3240681ca35b0fa0
Zekun Wang
670f270f3240681ca35b0fa1
Jiaheng Liu
claimed_verified
2025-01-05T23:03:31.306Z
65377c30e48353201e6fdda0
/avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
Jiaheng Liu
user
CheeryLJH
670f270f3240681ca35b0fa2
Xiaoshuai Song
670f270f3240681ca35b0fa3
Zhongyuan Peng
claimed_verified
2025-07-09T15:12:20.167Z
63299f93688ad82b783aaf20
/avatars/7c11e60e551ef1c62aa2862529e357f5.svg
zhongyuan peng
user
happzy2633
670f270f3240681ca35b0fa4
Ken Deng
670f270f3240681ca35b0fa5
Chenchen Zhang
670f270f3240681ca35b0fa6
Jiakai Wang
670f270f3240681ca35b0fa7
Junran Peng
670f270f3240681ca35b0fa8
Ge Zhang
claimed_verified
2024-10-30T09:55:40.883Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
670f270f3240681ca35b0fa9
Hangyu Guo
670f270f3240681ca35b0faa
Zhaoxiang Zhang
670f270f3240681ca35b0fab
Wenbo Su
670f270f3240681ca35b0fac
Bo Zheng
MTU-Bench: A Multi-granularity Tool-Use Benchmark for Large Language
  Models
Large Language Models (LLMs) have displayed massive improvements in reasoning
and decision-making skills and can hold natural conversations with users.
Recently, many tool-use benchmark datasets have been proposed. However,
existing datasets have the following limitations: (1). Insufficient evaluation
scenarios (e.g., only cover limited tool-use scenes). (2). Extensive evaluation
costs (e.g., GPT API costs). To address these limitations, in this work, we
propose a multi-granularity tool-use benchmark for large language models called
MTU-Bench. For the "multi-granularity" property, our MTU-Bench covers five tool
usage scenes (i.e., single-turn and single-tool, single-turn and multiple-tool,
multiple-turn and single-tool, multiple-turn and multiple-tool, and
out-of-distribution tasks). Besides, all evaluation metrics of our MTU-Bench
are based on the prediction results and the ground truth without using any GPT
or human evaluation metrics. Moreover, our MTU-Bench is collected by
transforming existing high-quality datasets to simulate real-world tool usage
scenarios, and we also propose an instruction dataset called MTU-Instruct data
to enhance the tool-use abilities of existing LLMs. Comprehensive experimental
results demonstrate the effectiveness of our MTU-Bench. Code and data will be
released at https: //github.com/MTU-Bench-Team/MTU-Bench.git.
A multi-granularity tool-use benchmark, MTU-Bench, is proposed to evaluate large language models across various tool usage scenarios, costing effectively without relying on GPT or human metrics.
Large Language Models LLMs tool-use benchmark MTU-Bench single-turn single-tool multiple-turn multiple-tool out-of-distribution tasks prediction results ground truth MTU-Instruct real-world tool usage scenarios