FML-bench: A Benchmark for Automatic ML Research Agents Highlighting the
  Importance of Exploration Breadth
Large language models (LLMs) have sparked growing interest in automatic
machine learning research agents. Among them, agents capable of autonomously
proposing ideas and conducting machine learning experiments are particularly
promising, as they maximize research automation and accelerate scientific
progress by iteratively refining ideas based on experimental results. However,
comprehensively evaluating such agents remains challenging. Existing benchmarks
tend to overemphasize engineering aspects while neglecting academic rigor,
creating barriers that obscure a clear assessment of an agent's scientific
capabilities in machine learning research. They also suffer from limited task
diversity, an overemphasis on application-oriented tasks over fundamental
research problems, and limited scalability to realistic research settings. To
address these limitations, we introduce FML-bench, a benchmark designed to
evaluate automatic machine learning research agents on 8 diverse and
fundamental machine learning research problems. It reduces coding burden,
emphasizes fundamental problems rather than specific use cases, offers high
task diversity, and is extensible to real-world machine learning GitHub
repositories. Furthermore, we present a unified evaluation framework with five
complementary metrics, designed to comprehensively assess agent performance on
our benchmark. We evaluate state-of-the-art automatic research agents on
FML-bench, and find that agents employing broad research exploration strategies
outperform those focusing on narrow but deep exploration. These findings
suggest that emphasizing the breadth of exploration may lead to more effective
research outcomes than focusing solely on incremental refinement. Our benchmark
is available at https://github.com/qrzou/FML-bench.
68ee45dd9b77b5223f6661d1
Qiran Zou
claimed_verified
2025-10-14T14:24:08.753Z
6763dc97d63e4b348ea3936c
/avatars/0f0567dcaa14276a8c2cb2bdfcc7ec80.svg
Qiran Zou
user
qiranzou
68ee45dd9b77b5223f6661d2
Hou Hei Lam
68ee45dd9b77b5223f6661d3
Wenhao Zhao
68ee45dd9b77b5223f6661d4
Yiming Tang
68ee45dd9b77b5223f6661d5
Tingting Chen
68ee45dd9b77b5223f6661d6
Samson Yu
68ee45dd9b77b5223f6661d7
Tianyi Zhang
68ee45dd9b77b5223f6661d8
Chang Liu
68ee45dd9b77b5223f6661d9
Xiangyang Ji
68ee45dd9b77b5223f6661da
Dianbo Liu
FML-bench: A Benchmark for Automatic ML Research Agents Highlighting the
  Importance of Exploration Breadth
Large language models (LLMs) have sparked growing interest in automatic
machine learning research agents. Among them, agents capable of autonomously
proposing ideas and conducting machine learning experiments are particularly
promising, as they maximize research automation and accelerate scientific
progress by iteratively refining ideas based on experimental results. However,
comprehensively evaluating such agents remains challenging. Existing benchmarks
tend to overemphasize engineering aspects while neglecting academic rigor,
creating barriers that obscure a clear assessment of an agent's scientific
capabilities in machine learning research. They also suffer from limited task
diversity, an overemphasis on application-oriented tasks over fundamental
research problems, and limited scalability to realistic research settings. To
address these limitations, we introduce FML-bench, a benchmark designed to
evaluate automatic machine learning research agents on 8 diverse and
fundamental machine learning research problems. It reduces coding burden,
emphasizes fundamental problems rather than specific use cases, offers high
task diversity, and is extensible to real-world machine learning GitHub
repositories. Furthermore, we present a unified evaluation framework with five
complementary metrics, designed to comprehensively assess agent performance on
our benchmark. We evaluate state-of-the-art automatic research agents on
FML-bench, and find that agents employing broad research exploration strategies
outperform those focusing on narrow but deep exploration. These findings
suggest that emphasizing the breadth of exploration may lead to more effective
research outcomes than focusing solely on incremental refinement. Our benchmark
is available at https://github.com/qrzou/FML-bench.
FML-bench evaluates automatic machine learning research agents on diverse fundamental problems using a unified framework with multiple metrics, highlighting the importance of broad exploration strategies.
large language models automatic machine learning research agents FML-bench evaluation framework research exploration strategies