MSC-Bench: A Rigorous Benchmark for Multi-Server Tool Orchestration
We introduce MSC-Bench, a large-scale benchmark for evaluating multi-hop,
end-to-end tool orchestration by LLM agents in a hierarchical Model-Context
Protocol (MCP) ecosystem. Existing benchmarks often evaluate tools in
isolation, ignoring challenges such as functional overlap and cross-server
orchestration, leading to overly optimistic assessments. MSC-Bench addresses
these gaps by constructing ground truth through 'equal function sets', allowing
objective metrics such as F1 score and reducing the dependency on
LLM-as-a-judge evaluation. Organized as a five-level curriculum, it
systematically tests agent capabilities from single-tool orchestration to
complex cross-server planning, and robustness to out-of-scope requests.
Experiments reveal that rigid hierarchies can hinder performance without
co-designed strategies, and even state-of-the-art agents exhibit systemic
weaknesses in robustness. MSC-Bench provides a diagnostic framework to expose
these limitations and guide the development of more capable and efficient
tool-using agents. The benchmark and resources are publicly available at
https://github.com/snooow1029/MSC_Bench.
68fb79986cdff8b857f46b30
Jia-Kai Dong
claimed_verified
2025-10-24T15:57:36.735Z
67ecfe11534d68f5a87834c2
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/ZnmCQzju-Okic9XdvJEvN.png
董家愷
user
Snooow1029
68fb79986cdff8b857f46b31
I-Wei Huang
68fb79986cdff8b857f46b32
Chun-Tin Wu
68fb79986cdff8b857f46b33
Yi-Tien Tsai
MSC-Bench: A Rigorous Benchmark for Multi-Server Tool Orchestration
We introduce MSC-Bench, a large-scale benchmark for evaluating multi-hop,
end-to-end tool orchestration by LLM agents in a hierarchical Model-Context
Protocol (MCP) ecosystem. Existing benchmarks often evaluate tools in
isolation, ignoring challenges such as functional overlap and cross-server
orchestration, leading to overly optimistic assessments. MSC-Bench addresses
these gaps by constructing ground truth through 'equal function sets', allowing
objective metrics such as F1 score and reducing the dependency on
LLM-as-a-judge evaluation. Organized as a five-level curriculum, it
systematically tests agent capabilities from single-tool orchestration to
complex cross-server planning, and robustness to out-of-scope requests.
Experiments reveal that rigid hierarchies can hinder performance without
co-designed strategies, and even state-of-the-art agents exhibit systemic
weaknesses in robustness. MSC-Bench provides a diagnostic framework to expose
these limitations and guide the development of more capable and efficient
tool-using agents. The benchmark and resources are publicly available at
https://github.com/snooow1029/MSC_Bench.
MSC-Bench evaluates multi-hop tool orchestration by LLM agents in a hierarchical ecosystem, addressing challenges like functional overlap and cross-server planning with a five-level curriculum and objective metrics.
LLM agents hierarchical Model-Context Protocol multi-hop tool orchestration equal function sets F1 score out-of-scope requests tool-using agents