The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT
  Improvements
Rapid advancements in large language models (LLMs) have the potential to
assist in scientific progress. A critical capability toward this endeavor is
the ability to reproduce existing work. To evaluate the ability of AI agents to
reproduce results in an active research area, we introduce the Automated LLM
Speedrunning Benchmark, leveraging the research community contributions on the
NanoGPT speedrun, a competition to train a GPT-2 model in the shortest time.
Each of the 19 speedrun tasks provides the agent with the previous records
training script, optionally paired with one of three hint formats, ranging from
pseudocode to paper-like descriptions of the new records improvements. Records
execute quickly by design and speedrun improvements encompass diverse
code-level changes, ranging from high-level algorithmic advancements to
hardware-aware optimizations. These features make the benchmark both accessible
and realistic for the frontier problem of improving LLM training. We find that
recent reasoning LLMs combined with SoTA scaffolds struggle to reimplement
already-known innovations in our benchmark, even when given detailed hints. Our
benchmark thus provides a simple, non-saturated measure of an LLMs ability to
automate scientific reproduction, a necessary (but not sufficient) skill for an
autonomous research agent.
686229249e7509383d29abd0
Bingchen Zhao
claimed_verified
2025-07-29T07:10:55.810Z
62dcd71075e9787ec5aa41ba
/avatars/f37ce036b76180ed0fa004f9c8c09363.svg
Bingchen Zhao
user
tennant
686229249e7509383d29abd1
Despoina Magka
claimed_verified
2025-07-01T10:14:25.980Z
673dc2345b07d224bff10173
/avatars/2b7b1491de77bc3193fd912de76e176d.svg
Despoina Magka
user
marlamagka
686229249e7509383d29abd2
Minqi Jiang
claimed_verified
2025-07-07T11:25:11.085Z
630368320547362a22a21798
/avatars/035b468584e43b4fbbf4c4aa1d2bbac7.svg
Minqi Jiang
user
minqi
686229249e7509383d29abd3
Xian Li
686229249e7509383d29abd4
Roberta Raileanu
686229249e7509383d29abd5
Tatiana Shavrina
686229249e7509383d29abd6
Jean-Christophe Gagnon-Audet
686229249e7509383d29abd7
Kelvin Niu
686229249e7509383d29abd8
Shagun Sodhani
686229249e7509383d29abd9
Michael Shvartsman
686229249e7509383d29abda
Andrei Lupu
686229249e7509383d29abdb
Alisia Lupidi
686229249e7509383d29abdc
Edan Toledo
686229249e7509383d29abdd
Karen Hambardzumyan
686229249e7509383d29abde
Martin Josifoski
686229249e7509383d29abdf
Thomas Foster
686229249e7509383d29abe0
Lucia Cipolina-Kun
686229249e7509383d29abe1
Abhishek Charnalia
686229249e7509383d29abe2
Derek Dunfield
686229249e7509383d29abe3
Alexander H. Miller
686229249e7509383d29abe4
Oisin Mac Aodha
686229249e7509383d29abe5
Jakob Foerster
686229249e7509383d29abe6
Yoram Bachrach
The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT
  Improvements
Rapid advancements in large language models (LLMs) have the potential to
assist in scientific progress. A critical capability toward this endeavor is
the ability to reproduce existing work. To evaluate the ability of AI agents to
reproduce results in an active research area, we introduce the Automated LLM
Speedrunning Benchmark, leveraging the research community contributions on the
NanoGPT speedrun, a competition to train a GPT-2 model in the shortest time.
Each of the 19 speedrun tasks provides the agent with the previous records
training script, optionally paired with one of three hint formats, ranging from
pseudocode to paper-like descriptions of the new records improvements. Records
execute quickly by design and speedrun improvements encompass diverse
code-level changes, ranging from high-level algorithmic advancements to
hardware-aware optimizations. These features make the benchmark both accessible
and realistic for the frontier problem of improving LLM training. We find that
recent reasoning LLMs combined with SoTA scaffolds struggle to reimplement
already-known innovations in our benchmark, even when given detailed hints. Our
benchmark thus provides a simple, non-saturated measure of an LLMs ability to
automate scientific reproduction, a necessary (but not sufficient) skill for an
autonomous research agent.
An Automated LLM Speedrunning Benchmark evaluates AI agents' ability to reproduce scientific results by leveraging NanoGPT speedrun tasks, indicating that even recent reasoning LLMs struggle with re-implementing known improvements.
large language models LLMs AI agents Automated LLM Speedrunning Benchmark NanoGPT speedrun GPT-2 high-level algorithmic advancements hardware-aware optimizations