PaperBench: Evaluating AI's Ability to Replicate AI Research
We introduce PaperBench, a benchmark evaluating the ability of AI agents to
replicate state-of-the-art AI research. Agents must replicate 20 ICML 2024
Spotlight and Oral papers from scratch, including understanding paper
contributions, developing a codebase, and successfully executing experiments.
For objective evaluation, we develop rubrics that hierarchically decompose each
replication task into smaller sub-tasks with clear grading criteria. In total,
PaperBench contains 8,316 individually gradable tasks. Rubrics are co-developed
with the author(s) of each ICML paper for accuracy and realism. To enable
scalable evaluation, we also develop an LLM-based judge to automatically grade
replication attempts against rubrics, and assess our judge's performance by
creating a separate benchmark for judges. We evaluate several frontier models
on PaperBench, finding that the best-performing tested agent, Claude 3.5 Sonnet
(New) with open-source scaffolding, achieves an average replication score of
21.0\%. Finally, we recruit top ML PhDs to attempt a subset of PaperBench,
finding that models do not yet outperform the human baseline. We
https://github.com/openai/preparedness{open-source our code} to
facilitate future research in understanding the AI engineering capabilities of
AI agents.
67edf3d579018bf61e050435
Giulio Starace
admin_assigned
2025-04-03T13:57:27.629Z
627b8bc83974b0ed6b28db67
https://cdn-avatars.huggingface.co/v1/production/uploads/1665158475637-627b8bc83974b0ed6b28db67.jpeg
Giulio Starace
user
thesofakillers
67edf3d579018bf61e050436
Oliver Jaffe
67edf3d579018bf61e050437
Dane Sherburn
67edf3d579018bf61e050438
James Aung
admin_assigned
2025-04-03T13:57:49.748Z
6316bd602fd6e930e429b362
/avatars/28abc3ad3a00811da404f5d88dd25a69.svg
James Aung
user
james-aung
67edf3d579018bf61e050439
Jun Shern Chan
admin_assigned
2025-04-03T13:58:00.906Z
60db178afb288e95534e7909
/avatars/54a656506a66d06aeb3556c88a9a072a.svg
Chan Jun Shern
user
junshern
67edf3d579018bf61e05043a
Leon Maksin
admin_assigned
2025-04-03T13:58:13.107Z
632c9a1c1d303f5f9ad0685c
/avatars/d8828064fe0d89efed7594c5c9747078.svg
M
user
LeonM
67edf3d579018bf61e05043b
Rachel Dias
67edf3d579018bf61e05043c
Evan Mays
admin_assigned
2025-04-03T13:58:23.365Z
636d9548da13cd91f922e0d0
/avatars/488f6a4b12f20809d3a4c3d40ae82b53.svg
Evan Mays
user
evanmays
67edf3d579018bf61e05043d
Benjamin Kinsella
admin_assigned
2025-04-03T13:58:29.583Z
65397cb82a101a151da4836b
/avatars/dd9e03be4e06fd41b82cce97c9ca2d21.svg
Benjamin Kinsella
user
bjk127
67edf3d579018bf61e05043e
Wyatt Thompson
67edf3d579018bf61e05043f
Johannes Heidecke
67edf3d579018bf61e050440
Amelia Glaese
67edf3d579018bf61e050441
Tejal Patwardhan
admin_assigned
2025-04-03T13:58:46.470Z
64d4f504887f55fb6eedec74
/avatars/054fb826890adcb330f0e4cbca3ef7c4.svg
Tejal Patwardhan
user
tejalp
PaperBench: Evaluating AI's Ability to Replicate AI Research
We introduce PaperBench, a benchmark evaluating the ability of AI agents to
replicate state-of-the-art AI research. Agents must replicate 20 ICML 2024
Spotlight and Oral papers from scratch, including understanding paper
contributions, developing a codebase, and successfully executing experiments.
For objective evaluation, we develop rubrics that hierarchically decompose each
replication task into smaller sub-tasks with clear grading criteria. In total,
PaperBench contains 8,316 individually gradable tasks. Rubrics are co-developed
with the author(s) of each ICML paper for accuracy and realism. To enable
scalable evaluation, we also develop an LLM-based judge to automatically grade
replication attempts against rubrics, and assess our judge's performance by
creating a separate benchmark for judges. We evaluate several frontier models
on PaperBench, finding that the best-performing tested agent, Claude 3.5 Sonnet
(New) with open-source scaffolding, achieves an average replication score of
21.0\%. Finally, we recruit top ML PhDs to attempt a subset of PaperBench,
finding that models do not yet outperform the human baseline. We
https://github.com/openai/preparedness{open-source our code} to
facilitate future research in understanding the AI engineering capabilities of
AI agents.
PaperBench evaluates AI agents' ability to replicate state-of-the-art AI research by decomposing replication tasks into graded sub-tasks, using both LLM-based and human judges to assess performance.
benchmark ICML 2024 rubrics LLM-based judge agent replication Claude 3.5 Sonnet ML PhDs AI engineering capabilities