Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
Boyu Gou, Zanming Huang, Yuting Ning, Yu Gu, Michael Lin, Botao Yu, Andrei Kopanev, Weijian Qi, Yiheng Shu, Jiaman Wu, Chan Hee Song, Bernal Jimenez Gutierrez, Yifei Li, Zeyi Liao, Hanane Moussa, TIANSHU ZHANG, Jian Xie, Tianci Xue, Shijie Chen, Boyuan Zheng, Kai Zhang, Zhaowei Cai, Viktor Rozgic, Morteza Ziyadi, Huan Sun, Yu Su
Agentic search such as Deep Research systems—where large language models autonomously browse the web, synthesize information, and return citation-backed answers—represents a major shift in how users interact with web-scale information. While promising greater efficiency and cognitive offloading, the growing complexity and open-endedness of agentic search has outpaced existing evaluation benchmarks and methodologies, which largely assume short horizons and static answers. In this paper, we introduce Mind2Web 2, a benchmark of 100 realistic, high-quality, long-horizon tasks that require real-time web interaction and extensive information synthesis. To address the challenge of evaluating time-varying, multi-source answers, we propose a novel Agent-as-a-Judge framework. Our method leverages task-specific, tree-structured rubrics and rubric-based judge agents to automatically assess both factual correctness and source attribution with a high agreement with humans. We conduct a comprehensive evaluation of 9 frontier agentic search systems and human performance, and a detailed error analysis to draw insights for future development. Together, Mind2Web 2 and our evaluation framework provide a rigorous foundation for developing and benchmarking the next generation of trustworthy, high-capability agentic search systems.