GTA: A Benchmark for General Tool Agents
Significant focus has been placed on integrating large language models (LLMs)
with various tools in developing general-purpose agents. This poses a challenge
to LLMs' tool-use capabilities. However, there are evident gaps between
existing tool-use evaluations and real-world scenarios. Current evaluations
often use AI-generated queries, single-step tasks, dummy tools, and text-only
interactions, failing to reveal the agents' real-world problem-solving
abilities effectively. To address this, we propose GTA, a benchmark for General
Tool Agents, featuring three main aspects: (i) Real user queries: human-written
queries with simple real-world objectives but implicit tool-use, requiring the
LLM to reason the suitable tools and plan the solution steps. (ii) Real
deployed tools: an evaluation platform equipped with tools across perception,
operation, logic, and creativity categories to evaluate the agents' actual task
execution performance. (iii) Real multimodal inputs: authentic image files,
such as spatial scenes, web page screenshots, tables, code snippets, and
printed/handwritten materials, used as the query contexts to align with
real-world scenarios closely. We design 229 real-world tasks and executable
tool chains to evaluate mainstream LLMs. Our findings show that real-world user
queries are challenging for existing LLMs, with GPT-4 completing less than 50%
of the tasks and most LLMs achieving below 25%. This evaluation reveals the
bottlenecks in the tool-use capabilities of current LLMs in real-world
scenarios, which provides future direction for advancing general-purpose tool
agents. The code and dataset are available at
https://github.com/open-compass/GTA.
66909d0698c0444e3a7e408a
Jize Wang
claimed_verified
2024-08-13T07:39:49.961Z
66323843a6ef9d3afdf11fbb
/avatars/2fe9313634fa30dfa3ca36de5cd094f0.svg
Jize
user
Jize1
66909d0698c0444e3a7e408b
Zerun Ma
claimed_verified
2024-07-22T08:43:33.636Z
634fb993685e43e02d2544a2
https://cdn-avatars.huggingface.co/v1/production/uploads/634fb993685e43e02d2544a2/L9MuXYlXMBuQgnwXd4lCU.jpeg
mzr1996
user
mzr1996
66909d0698c0444e3a7e408c
Yining Li
admin_assigned
2024-07-12T09:58:52.289Z
649988726677f66c2b486392
/avatars/b649a77370660e129726e29504daba34.svg
Yining Li
user
ly015
66909d0698c0444e3a7e408d
Songyang Zhang
admin_assigned
2024-07-12T09:57:07.530Z
630716d11801ecc7d2595021
/avatars/2d36a880ce4a3cf7efc5ff3987dbeaf3.svg
Songyang Zhang
user
zsytony
66909d0698c0444e3a7e408e
Cailian Chen
66909d0698c0444e3a7e408f
Kai Chen
66909d0698c0444e3a7e4090
Xinyi Le
GTA: A Benchmark for General Tool Agents
Significant focus has been placed on integrating large language models (LLMs)
with various tools in developing general-purpose agents. This poses a challenge
to LLMs' tool-use capabilities. However, there are evident gaps between
existing tool-use evaluations and real-world scenarios. Current evaluations
often use AI-generated queries, single-step tasks, dummy tools, and text-only
interactions, failing to reveal the agents' real-world problem-solving
abilities effectively. To address this, we propose GTA, a benchmark for General
Tool Agents, featuring three main aspects: (i) Real user queries: human-written
queries with simple real-world objectives but implicit tool-use, requiring the
LLM to reason the suitable tools and plan the solution steps. (ii) Real
deployed tools: an evaluation platform equipped with tools across perception,
operation, logic, and creativity categories to evaluate the agents' actual task
execution performance. (iii) Real multimodal inputs: authentic image files,
such as spatial scenes, web page screenshots, tables, code snippets, and
printed/handwritten materials, used as the query contexts to align with
real-world scenarios closely. We design 229 real-world tasks and executable
tool chains to evaluate mainstream LLMs. Our findings show that real-world user
queries are challenging for existing LLMs, with GPT-4 completing less than 50%
of the tasks and most LLMs achieving below 25%. This evaluation reveals the
bottlenecks in the tool-use capabilities of current LLMs in real-world
scenarios, which provides future direction for advancing general-purpose tool
agents. The code and dataset are available at
https://github.com/open-compass/GTA.
GTA benchmark evaluates LLMs' tool-use capabilities with real user queries, deployed tools, and multimodal inputs, revealing significant challenges and identifying bottlenecks.
General Tool Agents real user queries real deployed tools real multimodal inputs perceptual tools operational tools logical tools creative tools