Establishing Best Practices in Building Rigorous Agentic Benchmarks
Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun, Andy Zhang, Shu Liu, Sasha Cui, Sayash Kapoor, Shayne Longpre, Kevin Meng, Rebecca Weiss, Fazl Barez, Rahul Gupta, Jwala Dhamala, Jacob Merizian, Mario Giulianelli, Harry Coppock, Cozmin Ududec, Antony Kellermann, Jasjeet Sekhon, Jacob Steinhardt, Sarah Schwettmann, Arvind Narayanan, Matei A Zaharia, Ion Stoica, Percy Liang, Daniel Kang
Benchmarks are essential tools for quantitatively tracking progress in AI. As AI agents become increasingly capable of solving complex, real-world tasks, researchers and practitioners have introduced agentic benchmarks that evaluate agents based on task outcomes. However, we show that using outcome-based design suboptimally can misrepresent the true capabilities of agents. For example, SWE-bench-Verified uses insufficient test cases, while τ -bench counts empty responses as successful. Such flaws can lead to under- or overestimation agents’ performance by up to 100% in relative terms. To address this issue, we introduce the Agentic Benchmark Checklist (ABC), a set of benchmark development guidelines that we synthesized from our benchmark-building experience, a survey of best practices, and previously reported issues. When applied to CVE-bench, a benchmark with a particularly complex evaluation design, ABC reduces the performance overestimation by 33%.