MLE-Smith: Scaling MLE Tasks with Automated Multi-Agent Pipeline
While Language Models (LMs) have made significant progress in automating
machine learning engineering (MLE), the acquisition of high-quality MLE
training data is significantly constrained. Current MLE benchmarks suffer from
low scalability and limited applicability because they rely on static, manually
curated tasks, demanding extensive time and manual effort to produce. We
introduce MLE-Smith, a fully automated multi-agent pipeline, to transform raw
datasets into competition-style MLE challenges through an efficient
generate-verify-execute paradigm for scaling MLE tasks with verifiable quality,
real-world usability, and rich diversity. The proposed multi-agent pipeline in
MLE-Smith drives structured task design and standardized refactoring, coupled
with a hybrid verification mechanism that enforces strict structural rules and
high-level semantic soundness. It further validates empirical solvability and
real-world fidelity through interactive execution. We apply MLE-Smith to 224 of
real-world datasets and generate 606 tasks spanning multiple categories,
objectives, and modalities, demonstrating that MLE-Smith can work effectively
across a wide range of real-world datasets. Evaluation on the generated tasks
shows that the performance of eight mainstream and cutting-edge LLMs on
MLE-Smith tasks is strongly correlated with their performance on carefully
human-designed tasks, highlighting the effectiveness of the MLE-Smith to
scaling up MLE tasks, while maintaining task quality.
68e71d397ae125f9582e69a7
Rushi Qiang
claimed_verified
2025-10-09T06:50:16.218Z
6466e31a14e059dde8bbe4be
https://cdn-avatars.huggingface.co/v1/production/uploads/6466e31a14e059dde8bbe4be/D_DWGWEkhOnFdbbdCNP3N.jpeg
Rushi Qiang
user
Jerrycool
68e71d397ae125f9582e69a8
Yuchen Zhuang
68e71d397ae125f9582e69a9
Anikait Singh
68e71d397ae125f9582e69aa
Percy Liang
68e71d397ae125f9582e69ab
Chao Zhang
68e71d397ae125f9582e69ac
Sherry Yang
68e71d397ae125f9582e69ad
Bo Dai
MLE-Smith: Scaling MLE Tasks with Automated Multi-Agent Pipeline
While Language Models (LMs) have made significant progress in automating
machine learning engineering (MLE), the acquisition of high-quality MLE
training data is significantly constrained. Current MLE benchmarks suffer from
low scalability and limited applicability because they rely on static, manually
curated tasks, demanding extensive time and manual effort to produce. We
introduce MLE-Smith, a fully automated multi-agent pipeline, to transform raw
datasets into competition-style MLE challenges through an efficient
generate-verify-execute paradigm for scaling MLE tasks with verifiable quality,
real-world usability, and rich diversity. The proposed multi-agent pipeline in
MLE-Smith drives structured task design and standardized refactoring, coupled
with a hybrid verification mechanism that enforces strict structural rules and
high-level semantic soundness. It further validates empirical solvability and
real-world fidelity through interactive execution. We apply MLE-Smith to 224 of
real-world datasets and generate 606 tasks spanning multiple categories,
objectives, and modalities, demonstrating that MLE-Smith can work effectively
across a wide range of real-world datasets. Evaluation on the generated tasks
shows that the performance of eight mainstream and cutting-edge LLMs on
MLE-Smith tasks is strongly correlated with their performance on carefully
human-designed tasks, highlighting the effectiveness of the MLE-Smith to
scaling up MLE tasks, while maintaining task quality.
MLE-Smith automates the creation of high-quality, diverse MLE tasks from raw datasets using a multi-agent pipeline, improving scalability and maintaining task quality.
Language Models machine learning engineering MLE MLE-Smith multi-agent pipeline generate-verify-execute task design standardized refactoring hybrid verification structural rules semantic soundness empirical solvability real-world fidelity interactive execution