VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in
  Real-world Applications
As LLM-based agents are increasingly deployed in real-life scenarios,
existing benchmarks fail to capture their inherent complexity of handling
extensive information, leveraging diverse resources, and managing dynamic user
interactions. To address this gap, we introduce VitaBench, a challenging
benchmark that evaluates agents on versatile interactive tasks grounded in
real-world settings. Drawing from daily applications in food delivery, in-store
consumption, and online travel services, VitaBench presents agents with the
most complex life-serving simulation environment to date, comprising 66 tools.
Through a framework that eliminates domain-specific policies, we enable
flexible composition of these scenarios and tools, yielding 100 cross-scenario
tasks (main results) and 300 single-scenario tasks. Each task is derived from
multiple real user requests and requires agents to reason across temporal and
spatial dimensions, utilize complex tool sets, proactively clarify ambiguous
instructions, and track shifting user intent throughout multi-turn
conversations. Moreover, we propose a rubric-based sliding window evaluator,
enabling robust assessment of diverse solution pathways in complex environments
and stochastic interactions. Our comprehensive evaluation reveals that even the
most advanced models achieve only 30% success rate on cross-scenario tasks, and
less than 50% success rate on others. Overall, we believe VitaBench will serve
as a valuable resource for advancing the development of AI agents in practical
real-world applications. The code, dataset, and leaderboard are available at
https://vitabench.github.io/
68dc8ca34159d1f2418f9a7e
Wei He
claimed_verified
2025-10-01T10:22:01.757Z
66ecee857264238429a1211f
https://cdn-avatars.huggingface.co/v1/production/uploads/66ecee857264238429a1211f/TbuM7ToLBrSxDF8mOccpK.jpeg
Wei He
user
hewei2001
68dc8ca34159d1f2418f9a7f
Yueqing Sun
68dc8ca34159d1f2418f9a80
Hongyan Hao
68dc8ca34159d1f2418f9a81
Xueyuan Hao
68dc8ca34159d1f2418f9a82
Zhikang Xia
68dc8ca34159d1f2418f9a83
Qi Gu
68dc8ca34159d1f2418f9a84
Chengcheng Han
68dc8ca34159d1f2418f9a85
Dengchang Zhao
68dc8ca34159d1f2418f9a86
Hui Su
68dc8ca34159d1f2418f9a87
Kefeng Zhang
68dc8ca34159d1f2418f9a88
Man Gao
68dc8ca34159d1f2418f9a89
Xi Su
68dc8ca34159d1f2418f9a8a
Xiaodong Cai
68dc8ca34159d1f2418f9a8b
Xunliang Cai
68dc8ca34159d1f2418f9a8c
Yu Yang
68dc8ca34159d1f2418f9a8d
Yunke Zhao
claimed_verified
2025-10-15T07:08:17.623Z
64268def7a0d3f02acd4bdd4
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/exl68RpkgSPyhfBYZG3SJ.png
Corsky zhao
user
Corsky
VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in
  Real-world Applications
As LLM-based agents are increasingly deployed in real-life scenarios,
existing benchmarks fail to capture their inherent complexity of handling
extensive information, leveraging diverse resources, and managing dynamic user
interactions. To address this gap, we introduce VitaBench, a challenging
benchmark that evaluates agents on versatile interactive tasks grounded in
real-world settings. Drawing from daily applications in food delivery, in-store
consumption, and online travel services, VitaBench presents agents with the
most complex life-serving simulation environment to date, comprising 66 tools.
Through a framework that eliminates domain-specific policies, we enable
flexible composition of these scenarios and tools, yielding 100 cross-scenario
tasks (main results) and 300 single-scenario tasks. Each task is derived from
multiple real user requests and requires agents to reason across temporal and
spatial dimensions, utilize complex tool sets, proactively clarify ambiguous
instructions, and track shifting user intent throughout multi-turn
conversations. Moreover, we propose a rubric-based sliding window evaluator,
enabling robust assessment of diverse solution pathways in complex environments
and stochastic interactions. Our comprehensive evaluation reveals that even the
most advanced models achieve only 30% success rate on cross-scenario tasks, and
less than 50% success rate on others. Overall, we believe VitaBench will serve
as a valuable resource for advancing the development of AI agents in practical
real-world applications. The code, dataset, and leaderboard are available at
https://vitabench.github.io/
VitaBench is a benchmark for evaluating LLM-based agents in complex, real-world interactive tasks using a diverse set of tools and scenarios.
LLM-based agents VitaBench interactive tasks real-world settings food delivery in-store consumption online travel services life-serving simulation environment domain-specific policies flexible composition cross-scenario tasks single-scenario tasks real user requests temporal dimensions spatial dimensions complex tool sets ambiguous instructions shifting user intent multi-turn conversations rubric-based sliding window evaluator stochastic interactions