MLE-bench: Evaluating Machine Learning Agents on Machine Learning
  Engineering
We introduce MLE-bench, a benchmark for measuring how well AI agents perform
at machine learning engineering. To this end, we curate 75 ML
engineering-related competitions from Kaggle, creating a diverse set of
challenging tasks that test real-world ML engineering skills such as training
models, preparing datasets, and running experiments. We establish human
baselines for each competition using Kaggle's publicly available leaderboards.
We use open-source agent scaffolds to evaluate several frontier language models
on our benchmark, finding that the best-performing setup--OpenAI's o1-preview
with AIDE scaffolding--achieves at least the level of a Kaggle bronze medal in
16.9% of competitions. In addition to our main results, we investigate various
forms of resource scaling for AI agents and the impact of contamination from
pre-training. We open-source our benchmark code (github.com/openai/mle-bench/)
to facilitate future research in understanding the ML engineering capabilities
of AI agents.
67077d44503533522daed22f
Jun Shern Chan
67077d44503533522daed230
Neil Chowdhury
admin_assigned
2024-10-10T09:44:54.162Z
63ec754359a46fd5f4b81a2f
/avatars/aaa4a63b25bd03c366f855427ba9cad4.svg
Neil Chowdhury
user
nch0w
67077d44503533522daed231
Oliver Jaffe
67077d44503533522daed232
James Aung
67077d44503533522daed233
Dane Sherburn
67077d44503533522daed234
Evan Mays
admin_assigned
2024-10-10T09:44:33.433Z
636d9548da13cd91f922e0d0
/avatars/488f6a4b12f20809d3a4c3d40ae82b53.svg
Evan Mays
user
evanmays
67077d44503533522daed235
Giulio Starace
admin_assigned
2024-10-10T09:44:26.853Z
627b8bc83974b0ed6b28db67
https://cdn-avatars.huggingface.co/v1/production/uploads/1665158475637-627b8bc83974b0ed6b28db67.jpeg
Giulio Starace
user
thesofakillers
67077d44503533522daed236
Kevin Liu
67077d44503533522daed237
Leon Maksin
67077d44503533522daed238
Tejal Patwardhan
admin_assigned
2024-10-10T09:44:12.948Z
64d4f504887f55fb6eedec74
/avatars/054fb826890adcb330f0e4cbca3ef7c4.svg
Tejal Patwardhan
user
tejalp
67077d44503533522daed239
Lilian Weng
admin_assigned
2024-10-10T09:44:07.283Z
5fb2dc322a79d831217aa0cb
/avatars/c4bdc2f907443e34d75116730105510c.svg
Lilian Weng
user
lilianweng
67077d44503533522daed23a
Aleksander Mądry
MLE-bench: Evaluating Machine Learning Agents on Machine Learning
  Engineering
We introduce MLE-bench, a benchmark for measuring how well AI agents perform
at machine learning engineering. To this end, we curate 75 ML
engineering-related competitions from Kaggle, creating a diverse set of
challenging tasks that test real-world ML engineering skills such as training
models, preparing datasets, and running experiments. We establish human
baselines for each competition using Kaggle's publicly available leaderboards.
We use open-source agent scaffolds to evaluate several frontier language models
on our benchmark, finding that the best-performing setup--OpenAI's o1-preview
with AIDE scaffolding--achieves at least the level of a Kaggle bronze medal in
16.9% of competitions. In addition to our main results, we investigate various
forms of resource scaling for AI agents and the impact of contamination from
pre-training. We open-source our benchmark code (github.com/openai/mle-bench/)
to facilitate future research in understanding the ML engineering capabilities
of AI agents.
MLE-bench benchmark evaluates AI agents' machine learning engineering skills through curated Kaggle competitions, showing that the best setup can achieve bronze medal performance in some competitions.
machine learning engineering AI agents MLE-bench Kaggle training models preparing datasets running experiments human baselines OpenAI o1-preview AIDE scaffolding