MLE-bench: Evaluating Machine Learning Agents on Machine Learning
  Engineering
We introduce MLE-bench, a benchmark for measuring how well AI agents perform
at machine learning engineering. To this end, we curate 75 ML
engineering-related competitions from Kaggle, creating a diverse set of
challenging tasks that test real-world ML engineering skills such as training
models, preparing datasets, and running experiments. We establish human
baselines for each competition using Kaggle's publicly available leaderboards.
We use open-source agent scaffolds to evaluate several frontier language models
on our benchmark, finding that the best-performing setup--OpenAI's o1-preview
with AIDE scaffolding--achieves at least the level of a Kaggle bronze medal in
16.9% of competitions. In addition to our main results, we investigate various
forms of resource scaling for AI agents and the impact of contamination from
pre-training. We open-source our benchmark code (github.com/openai/mle-bench/)
to facilitate future research in understanding the ML engineering capabilities
of AI agents.
67077d44503533522daed22f
Jun Shern Chan
67077d44503533522daed230
Neil Chowdhury
admin_assigned
2024-10-10T09:44:54.162Z
63ec754359a46fd5f4b81a2f
/avatars/aaa4a63b25bd03c366f855427ba9cad4.svg
Neil Chowdhury
user
nch0w
67077d44503533522daed231
Oliver Jaffe
67077d44503533522daed232
James Aung
67077d44503533522daed233
Dane Sherburn
67077d44503533522daed234
Evan Mays
admin_assigned
2024-10-10T09:44:33.433Z
636d9548da13cd91f922e0d0
/avatars/488f6a4b12f20809d3a4c3d40ae82b53.svg
Evan Mays
user
evanmays
67077d44503533522daed235
Giulio Starace
admin_assigned
2024-10-10T09:44:26.853Z
627b8bc83974b0ed6b28db67
https://cdn-avatars.huggingface.co/v1/production/uploads/1665158475637-627b8bc83974b0ed6b28db67.jpeg
Giulio Starace
user
thesofakillers
67077d44503533522daed236
Kevin Liu
67077d44503533522daed237
Leon Maksin
67077d44503533522daed238
Tejal Patwardhan
admin_assigned
2024-10-10T09:44:12.948Z
64d4f504887f55fb6eedec74
/avatars/054fb826890adcb330f0e4cbca3ef7c4.svg
Tejal Patwardhan
user
tejalp
67077d44503533522daed239
Lilian Weng
admin_assigned
2024-10-10T09:44:07.283Z
5fb2dc322a79d831217aa0cb
/avatars/c4bdc2f907443e34d75116730105510c.svg
Lilian Weng
user
lilianweng
67077d44503533522daed23a
Aleksander Mądry
MLE-bench: Evaluating Machine Learning Agents on Machine Learning
  Engineering
We introduce MLE-bench, a benchmark for measuring how well AI agents perform
at machine learning engineering. To this end, we curate 75 ML
engineering-related competitions from Kaggle, creating a diverse set of
challenging tasks that test real-world ML engineering skills such as training
models, preparing datasets, and running experiments. We establish human
baselines for each competition using Kaggle's publicly available leaderboards.
We use open-source agent scaffolds to evaluate several frontier language models
on our benchmark, finding that the best-performing setup--OpenAI's o1-preview
with AIDE scaffolding--achieves at least the level of a Kaggle bronze medal in
16.9% of competitions. In addition to our main results, we investigate various
forms of resource scaling for AI agents and the impact of contamination from
pre-training. We open-source our benchmark code (github.com/openai/mle-bench/)
to facilitate future research in understanding the ML engineering capabilities
of AI agents.
MLE-bench benchmark AI agents machine learning engineering ML engineering-related competitions Kaggle training models preparing datasets running experiments human baselines leaderboards OpenAI's o1-preview AIDE scaffolding bronze medal resource language models resource scaling contamination from pre-training open-source benchmark code