LiveMCP-101: Stress Testing and Diagnosing MCP-enabled Agents on
  Challenging Queries
Tool calling has emerged as a critical capability for AI agents to interact
with the real world and solve complex tasks. While the Model Context Protocol
(MCP) provides a powerful standardized framework for tool integration, there is
a significant gap in benchmarking how well AI agents can effectively solve
multi-step tasks using diverse MCP tools in realistic, dynamic scenarios. In
this work, we present LiveMCP-101, a benchmark of 101 carefully curated
real-world queries, refined through iterative LLM rewriting and manual review,
that require coordinated use of multiple MCP tools including web search, file
operations, mathematical reasoning, and data analysis. Moreover, we introduce a
novel evaluation approach that leverages ground-truth execution plans rather
than raw API outputs, better reflecting the evolving nature of real-world
environments. Experiments show that even frontier LLMs achieve a success rate
below 60\%, highlighting major challenges in tool orchestration. Detailed
ablations and error analysis further reveal distinct failure modes and
inefficiencies in token usage, pointing to concrete directions for advancing
current models. LiveMCP-101 sets a rigorous standard for evaluating real-world
agent capabilities, advancing toward autonomous AI systems that reliably
execute complex tasks through tool use.
68a80f2939413c456c05b1b6
Ming Yin
claimed_verified
2025-08-22T07:21:36.904Z
66d4f80bbb76fb26ee6298cf
/avatars/b341445ecc7f9b1c63b4bcba94d83bd0.svg
Ming Yin
user
Kevin355
68a80f2939413c456c05b1b7
Dinghan Shen
admin_assigned
2025-08-22T08:29:47.464Z
66a6c378248478833bb38d26
/avatars/c8930d93645f106a0e73bff0c0b7da4c.svg
Dinghan Shen
user
dinghanshen
68a80f2939413c456c05b1b8
Silei Xu
admin_assigned
2025-08-22T08:29:53.290Z
6594b1bb57a556fbe162915e
https://cdn-avatars.huggingface.co/v1/production/uploads/6594b1bb57a556fbe162915e/WuYxqbbvaJaT-xsk5KhoT.jpeg
Silei Xu
user
sileixu
68a80f2939413c456c05b1b9
Jianbing Han
claimed_verified
2025-08-25T08:21:34.034Z
672d9b31857c221d2629e284
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/TLrcd554-qskfU1fW8SZU.png
Jianbing Han
user
Kian2527
68a80f2939413c456c05b1ba
Sixun Dong
admin_assigned
2025-08-22T08:30:06.275Z
653281ba1995cee54ac7ac4d
/avatars/077197d7b00502216b8edb1c87544791.svg
Sixun Dong
user
Ironieser
68a80f2939413c456c05b1bb
Mian Zhang
68a80f2939413c456c05b1bc
Yebowen Hu
admin_assigned
2025-08-22T08:30:12.076Z
61638ee4877577e50b485cd4
https://cdn-avatars.huggingface.co/v1/production/uploads/1633914587203-noauth.jpeg
Yebowen Hu
user
huuuyeah
68a80f2939413c456c05b1bd
Shujian Liu
admin_assigned
2025-08-22T08:30:17.542Z
637b90c9086af1cb122e6527
/avatars/7ea8bb105b33a13e8d0e2e0bc80792ac.svg
shujian liu
user
learning-rate-hunter
68a80f2939413c456c05b1be
Simin Ma
admin_assigned
2025-08-22T08:30:23.792Z
67063b3b1f78930da39202b8
/avatars/e1665a62e7b660eb959c8ad61b87dda9.svg
Simin Ma
user
masimaaa
68a80f2939413c456c05b1bf
Song Wang
claimed_verified
2025-08-26T09:50:41.479Z
624be9911fb0c140d4e8768a
/avatars/fe8a253f2f4fcdf9c60eead7bb90180a.svg
SongW
user
songwang41
68a80f2939413c456c05b1c0
Sathish Reddy Indurthi
admin_assigned
2025-08-22T08:30:29.779Z
631059bea951093de05aedbd
/avatars/f44ebb00a09776c29dd6ec96dcd91f24.svg
Sathish Reddy Indurthi
user
isathish
68a80f2939413c456c05b1c1
Xun Wang
68a80f2939413c456c05b1c2
Yiran Chen
68a80f2939413c456c05b1c3
Kaiqiang Song
admin_assigned
2025-08-22T08:30:35.767Z
63a261f13c003e409322c191
/avatars/dc4be6ce1983d0d32620275df124fe45.svg
Kaiqiang Song
user
kqsong
LiveMCP-101: Stress Testing and Diagnosing MCP-enabled Agents on
  Challenging Queries
Tool calling has emerged as a critical capability for AI agents to interact
with the real world and solve complex tasks. While the Model Context Protocol
(MCP) provides a powerful standardized framework for tool integration, there is
a significant gap in benchmarking how well AI agents can effectively solve
multi-step tasks using diverse MCP tools in realistic, dynamic scenarios. In
this work, we present LiveMCP-101, a benchmark of 101 carefully curated
real-world queries, refined through iterative LLM rewriting and manual review,
that require coordinated use of multiple MCP tools including web search, file
operations, mathematical reasoning, and data analysis. Moreover, we introduce a
novel evaluation approach that leverages ground-truth execution plans rather
than raw API outputs, better reflecting the evolving nature of real-world
environments. Experiments show that even frontier LLMs achieve a success rate
below 60\%, highlighting major challenges in tool orchestration. Detailed
ablations and error analysis further reveal distinct failure modes and
inefficiencies in token usage, pointing to concrete directions for advancing
current models. LiveMCP-101 sets a rigorous standard for evaluating real-world
agent capabilities, advancing toward autonomous AI systems that reliably
execute complex tasks through tool use.
LiveMCP-101 benchmarks AI agents' ability to use multiple tools in real-world scenarios, revealing challenges in tool orchestration and inefficiencies in token usage.
Model Context Protocol MCP LiveMCP-101 real-world queries LLM rewriting web search file operations mathematical reasoning data analysis ground-truth execution plans token usage