MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP
  Use
MCP standardizes how LLMs interact with external systems, forming the
foundation for general agents. However, existing MCP benchmarks remain narrow
in scope: they focus on read-heavy tasks or tasks with limited interaction
depth, and fail to capture the complexity and realism of real-world workflows.
To address this gap, we propose MCPMark, a benchmark designed to evaluate MCP
use in a more realistic and comprehensive manner. It consists of 127
high-quality tasks collaboratively created by domain experts and AI agents.
Each task begins with a curated initial state and includes a programmatic
script for automatic verification. These tasks demand richer and more diverse
interactions with the environment, involving a broad range of create, read,
update, and delete (CRUD) operations. We conduct a comprehensive evaluation of
cutting-edge LLMs using a minimal agent framework that operates in a
tool-calling loop. Empirical results show that the best-performing model,
gpt-5-medium, reaches only 52.56\% pass@1 and 33.86\% pass^4, while other
widely regarded strong models, including claude-sonnet-4 and o3, fall below
30\% pass@1 and 15\% pass^4. On average, LLMs require 16.2 execution
turns and 17.4 tool calls per task, significantly surpassing those in
previous MCP benchmarks and highlighting the stress-testing nature of MCPMark.
68dc9eff4159d1f2418f9b97
Zijian Wu
claimed_verified
2025-10-01T10:21:21.979Z
626d268d5f7327906f05cad1
/avatars/18bda74612a3ee63a17f991bcc695106.svg
Zijian Wu
user
Jakumetsu
68dc9eff4159d1f2418f9b98
Xiangyan Liu
68dc9eff4159d1f2418f9b99
Xinyuan Zhang
68dc9eff4159d1f2418f9b9a
Lingjun Chen
claimed_verified
2025-10-01T15:29:18.308Z
65d95ea62a6348b052519214
/avatars/b1d2fc97ad5df7cf187be48daa0e1f34.svg
chen
user
chenlingjun
68dc9eff4159d1f2418f9b9b
Fanqing Meng
68dc9eff4159d1f2418f9b9c
Lingxiao Du
claimed_verified
2025-10-01T10:21:32.767Z
666fe1a5b07525f0bde69c27
/avatars/bb98ab0b974c8fe011739baa8dadd91a.svg
Lingxiao Du
user
Cierra0506
68dc9eff4159d1f2418f9b9d
Yiran Zhao
claimed_verified
2025-10-02T13:56:23.095Z
64802face9ff472e30dc1ceb
https://cdn-avatars.huggingface.co/v1/production/uploads/64802face9ff472e30dc1ceb/bcwTlgpaUrU7m2RMB5zCc.png
Yiran Zhao
user
Yiran0924
68dc9eff4159d1f2418f9b9e
Fanshi Zhang
68dc9eff4159d1f2418f9b9f
Yaoqi Ye
68dc9eff4159d1f2418f9ba0
Jiawei Wang
claimed_verified
2025-10-02T03:08:55.923Z
64060b49a577649430bf6974
/avatars/74d0d6ed656b593e4c101b09edf18c7a.svg
Jiawei Wang
user
Jarvis1111
68dc9eff4159d1f2418f9ba1
Zirui Wang
68dc9eff4159d1f2418f9ba2
Jinjie Ni
68dc9eff4159d1f2418f9ba3
Yufan Yang
68dc9eff4159d1f2418f9ba4
Arvin Xu
claimed_verified
2025-10-01T15:29:20.221Z
64189e45d13ffa40812ac44c
/avatars/472ac989846f7c371067bfbeceeffb05.svg
Arvin Xu
user
arvinxx
68dc9eff4159d1f2418f9ba5
Michael Qizhe Shieh
MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP
  Use
MCP standardizes how LLMs interact with external systems, forming the
foundation for general agents. However, existing MCP benchmarks remain narrow
in scope: they focus on read-heavy tasks or tasks with limited interaction
depth, and fail to capture the complexity and realism of real-world workflows.
To address this gap, we propose MCPMark, a benchmark designed to evaluate MCP
use in a more realistic and comprehensive manner. It consists of 127
high-quality tasks collaboratively created by domain experts and AI agents.
Each task begins with a curated initial state and includes a programmatic
script for automatic verification. These tasks demand richer and more diverse
interactions with the environment, involving a broad range of create, read,
update, and delete (CRUD) operations. We conduct a comprehensive evaluation of
cutting-edge LLMs using a minimal agent framework that operates in a
tool-calling loop. Empirical results show that the best-performing model,
gpt-5-medium, reaches only 52.56\% pass@1 and 33.86\% pass^4, while other
widely regarded strong models, including claude-sonnet-4 and o3, fall below
30\% pass@1 and 15\% pass^4. On average, LLMs require 16.2 execution
turns and 17.4 tool calls per task, significantly surpassing those in
previous MCP benchmarks and highlighting the stress-testing nature of MCPMark.
MCPMark is a comprehensive benchmark for evaluating MCP use in real-world workflows, featuring diverse tasks that require richer interactions with the environment, and reveals that current LLMs perform poorly on these tasks.
MCP LLMs general agents MCP benchmarks MCPMark high-quality tasks domain experts AI agents initial state programmatic script automatic verification CRUD operations minimal agent framework tool-calling loop gpt-5-medium claude-sonnet-4 o3 pass@1 pass^4 execution turns tool calls