EXP-Bench: Can AI Conduct AI Research Experiments?
Automating AI research holds immense potential for accelerating scientific
progress, yet current AI agents struggle with the complexities of rigorous,
end-to-end experimentation. We introduce EXP-Bench, a novel benchmark designed
to systematically evaluate AI agents on complete research experiments sourced
from influential AI publications. Given a research question and incomplete
starter code, EXP-Bench challenges AI agents to formulate hypotheses, design
and implement experimental procedures, execute them, and analyze results. To
enable the creation of such intricate and authentic tasks with high-fidelity,
we design a semi-autonomous pipeline to extract and structure crucial
experimental details from these research papers and their associated
open-source code. With the pipeline, EXP-Bench curated 461 AI research tasks
from 51 top-tier AI research papers. Evaluations of leading LLM-based agents,
such as OpenHands and IterativeAgent on EXP-Bench demonstrate partial
capabilities: while scores on individual experimental aspects such as design or
implementation correctness occasionally reach 20-35%, the success rate for
complete, executable experiments was a mere 0.5%. By identifying these
bottlenecks and providing realistic step-by-step experiment procedures,
EXP-Bench serves as a vital tool for future AI agents to improve their ability
to conduct AI research experiments. EXP-Bench is open-sourced at
https://github.com/Just-Curieous/Curie/tree/main/benchmark/exp_bench.
683dfe8b4acfa22520c6a9ed
Patrick Tser Jern Kon
claimed_verified
2025-06-03T08:47:31.338Z
64b7111e17681d64b19cf95e
https://cdn-avatars.huggingface.co/v1/production/uploads/64b7111e17681d64b19cf95e/VHPfCUl1nBS3OMMVi96CR.jpeg
Patrick Kon
user
patkon
683dfe8b4acfa22520c6a9ee
Jiachen Liu
683dfe8b4acfa22520c6a9ef
Xinyi Zhu
683dfe8b4acfa22520c6a9f0
Qiuyi Ding
683dfe8b4acfa22520c6a9f1
Jingjia Peng
683dfe8b4acfa22520c6a9f2
Jiarong Xing
683dfe8b4acfa22520c6a9f3
Yibo Huang
683dfe8b4acfa22520c6a9f4
Yiming Qiu
683dfe8b4acfa22520c6a9f5
Jayanth Srinivasa
683dfe8b4acfa22520c6a9f6
Myungjin Lee
683dfe8b4acfa22520c6a9f7
Mosharaf Chowdhury
683dfe8b4acfa22520c6a9f8
Matei Zaharia
683dfe8b4acfa22520c6a9f9
Ang Chen
EXP-Bench: Can AI Conduct AI Research Experiments?
Automating AI research holds immense potential for accelerating scientific
progress, yet current AI agents struggle with the complexities of rigorous,
end-to-end experimentation. We introduce EXP-Bench, a novel benchmark designed
to systematically evaluate AI agents on complete research experiments sourced
from influential AI publications. Given a research question and incomplete
starter code, EXP-Bench challenges AI agents to formulate hypotheses, design
and implement experimental procedures, execute them, and analyze results. To
enable the creation of such intricate and authentic tasks with high-fidelity,
we design a semi-autonomous pipeline to extract and structure crucial
experimental details from these research papers and their associated
open-source code. With the pipeline, EXP-Bench curated 461 AI research tasks
from 51 top-tier AI research papers. Evaluations of leading LLM-based agents,
such as OpenHands and IterativeAgent on EXP-Bench demonstrate partial
capabilities: while scores on individual experimental aspects such as design or
implementation correctness occasionally reach 20-35%, the success rate for
complete, executable experiments was a mere 0.5%. By identifying these
bottlenecks and providing realistic step-by-step experiment procedures,
EXP-Bench serves as a vital tool for future AI agents to improve their ability
to conduct AI research experiments. EXP-Bench is open-sourced at
https://github.com/Just-Curieous/Curie/tree/main/benchmark/exp_bench.
EXP-Bench evaluates AI agents' end-to-end research experiment capabilities through curated tasks from top AI papers, highlighting current limitations.
LLM-based agents OpenHands IterativeAgent end-to-end experimentation