RExBench: Can coding agents autonomously implement AI research
  extensions?
Agents based on Large Language Models (LLMs) have shown promise for
performing sophisticated software engineering tasks autonomously. In addition,
there has been progress towards developing agents that can perform parts of the
research pipeline in machine learning and the natural sciences. We argue that
research extension and its implementation is a critical capability for such
systems, and introduce RExBench to support the evaluation of this capability.
RExBench is a benchmark consisting of 12 realistic research experiment
implementation tasks that aim to investigate research hypotheses that have not
previously been implemented. Each task is set up as an extension to an existing
research paper and codebase, accompanied by domain expert-written instructions.
RExBench is robust to data contamination, and supports an automatic evaluation
infrastructure that executes agent outputs to determine whether the success
criteria are met. We use this benchmark to evaluate nine LLM agents implemented
using three different frameworks: aider, Claude Code, and OpenHands. We find
that all agents evaluated fail to autonomously implement the majority of the
extensions. Although the success rate improves with additional human-written
hints, the best performance under this setting remains below 40%. This
indicates that current agents are still short of being able to handle realistic
research extension tasks without substantial human guidance.
6863f2bf7265b688e782627c
Nicholas Edwards
6863f2bf7265b688e782627d
Yukyung Lee
6863f2bf7265b688e782627e
Yujun
6863f2bf7265b688e782627f
Mao
6863f2bf7265b688e7826280
Yulu Qin
6863f2bf7265b688e7826281
Sebastian Schuster
6863f2bf7265b688e7826282
Najoung Kim
claimed_verified
2025-07-02T07:40:22.325Z
627c0b0a7e62b4bf5c37ea65
/avatars/7b459b6828bcdfd66bc3fbd922606d14.svg
Najoung Kim
user
najoungkim
RExBench: Can coding agents autonomously implement AI research
  extensions?
Agents based on Large Language Models (LLMs) have shown promise for
performing sophisticated software engineering tasks autonomously. In addition,
there has been progress towards developing agents that can perform parts of the
research pipeline in machine learning and the natural sciences. We argue that
research extension and its implementation is a critical capability for such
systems, and introduce RExBench to support the evaluation of this capability.
RExBench is a benchmark consisting of 12 realistic research experiment
implementation tasks that aim to investigate research hypotheses that have not
previously been implemented. Each task is set up as an extension to an existing
research paper and codebase, accompanied by domain expert-written instructions.
RExBench is robust to data contamination, and supports an automatic evaluation
infrastructure that executes agent outputs to determine whether the success
criteria are met. We use this benchmark to evaluate nine LLM agents implemented
using three different frameworks: aider, Claude Code, and OpenHands. We find
that all agents evaluated fail to autonomously implement the majority of the
extensions. Although the success rate improves with additional human-written
hints, the best performance under this setting remains below 40%. This
indicates that current agents are still short of being able to handle realistic
research extension tasks without substantial human guidance.
RExBench evaluates the capability of LLM agents to autonomously implement research extensions, finding that current agents require significant human guidance to succeed.
Large Language Models LLMs software engineering tasks research pipeline machine learning natural sciences research extension RExBench benchmark research experiment implementation tasks domain expert-written instructions automatic evaluation infrastructure success criteria aider Claude Code OpenHands