Fantastic Bugs and Where to Find Them in AI Benchmarks
Sang Truong, Yuheng Tu, Michael Hardy, Anka Reuel-Lamparth, Zeyu Tang, Jirayu Burapacheep, Jonathan Perera, Chibuike Uwakwe, Benjamin Domingue, Nick Haber, Sanmi Koyejo
Benchmarks are pivotal in driving progress in large language models, yet ambiguous questions, incorrect answer keys, and grading issues frequently undermine their reliability. Manually identifying and fixing issues among thousands of benchmark questions is not only infeasible but also a critical bottleneck for reliable evaluation. In this work, we introduce a scalable, theory-driven framework for systematic benchmark revision that leverages psychometric tools to flag problematic questions requiring expert review. We demonstrate that the No Free Lunch theorem applies directly to benchmark quality assessment: no detector can excel across all anomaly patterns, and effective detection requires prior anomaly knowledge. Furthermore, recognizing the high cost of LLM evaluations and the limited diversity of available LLMs, we assess each tool’s sensitivity to the number of model responses. Finally, across nine widely used benchmarks, our signals guide expert review to identify flawed questions with up to 84\% precision, offering an efficient, scalable framework for systematic benchmark revision.