TheMCPCompany: Creating General-purpose Agents with Task-specific Tools
Since the introduction of the Model Context Protocol (MCP), the number of
available tools for Large Language Models (LLMs) has increased significantly.
These task-specific tool sets offer an alternative to general-purpose tools
such as web browsers, while being easier to develop and maintain than GUIs.
However, current general-purpose agents predominantly rely on web browsers for
interacting with the environment. Here, we introduce TheMCPCompany, a benchmark
for evaluating tool-calling agents on tasks that involve interacting with
various real-world services. We use the REST APIs of these services to create
MCP servers, which include over 18,000 tools. We also provide manually
annotated ground-truth tools for each task. In our experiments, we use the
ground truth tools to show the potential of tool-calling agents for both
improving performance and reducing costs assuming perfect tool retrieval. Next,
we explore agent performance using tool retrieval to study the real-world
practicality of tool-based agents. While all models with tool retrieval perform
similarly or better than browser-based agents, smaller models cannot take full
advantage of the available tools through retrieval. On the other hand, GPT-5's
performance with tool retrieval is very close to its performance with
ground-truth tools. Overall, our work shows that the most advanced reasoning
models are effective at discovering tools in simpler environments, but
seriously struggle with navigating complex enterprise environments.
TheMCPCompany reveals that navigating tens of thousands of tools and combining
them in non-trivial ways to solve complex problems is still a challenging task
for current models and requires both better reasoning and better retrieval
models.
68f98627b9b2e4ae04673747
Reza Esfandiarpoor
68f98627b9b2e4ae04673748
Vishwas Suryanarayanan
68f98627b9b2e4ae04673749
Stephen H. Bach
68f98627b9b2e4ae0467374a
Vishal Chowdhary
68f98627b9b2e4ae0467374b
Anthony Aue
TheMCPCompany: Creating General-purpose Agents with Task-specific Tools
Since the introduction of the Model Context Protocol (MCP), the number of
available tools for Large Language Models (LLMs) has increased significantly.
These task-specific tool sets offer an alternative to general-purpose tools
such as web browsers, while being easier to develop and maintain than GUIs.
However, current general-purpose agents predominantly rely on web browsers for
interacting with the environment. Here, we introduce TheMCPCompany, a benchmark
for evaluating tool-calling agents on tasks that involve interacting with
various real-world services. We use the REST APIs of these services to create
MCP servers, which include over 18,000 tools. We also provide manually
annotated ground-truth tools for each task. In our experiments, we use the
ground truth tools to show the potential of tool-calling agents for both
improving performance and reducing costs assuming perfect tool retrieval. Next,
we explore agent performance using tool retrieval to study the real-world
practicality of tool-based agents. While all models with tool retrieval perform
similarly or better than browser-based agents, smaller models cannot take full
advantage of the available tools through retrieval. On the other hand, GPT-5's
performance with tool retrieval is very close to its performance with
ground-truth tools. Overall, our work shows that the most advanced reasoning
models are effective at discovering tools in simpler environments, but
seriously struggle with navigating complex enterprise environments.
TheMCPCompany reveals that navigating tens of thousands of tools and combining
them in non-trivial ways to solve complex problems is still a challenging task
for current models and requires both better reasoning and better retrieval
models.
TheMCPCompany evaluates tool-calling agents using REST APIs for interacting with real-world services, showing that advanced models perform well in simpler environments but struggle with complex enterprise environments.
Model Context Protocol Large Language Models tool-calling agents REST APIs ground-truth tools tool retrieval GPT-5 reasoning models retrieval models