ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm
  Engineering
How well do AI systems perform in algorithm engineering for hard optimization
problems in domains such as package-delivery routing, crew scheduling, factory
production planning, and power-grid balancing? We introduce ALE-Bench, a new
benchmark for evaluating AI systems on score-based algorithmic programming
contests. Drawing on real tasks from the AtCoder Heuristic Contests, ALE-Bench
presents optimization problems that are computationally hard and admit no known
exact solution. Unlike short-duration, pass/fail coding benchmarks, ALE-Bench
encourages iterative solution refinement over long time horizons. Our software
framework supports interactive agent architectures that leverage test-run
feedback and visualizations. Our evaluation of frontier LLMs revealed that
while they demonstrate high performance on specific problems, a notable gap
remains compared to humans in terms of consistency across problems and
long-horizon problem-solving capabilities. This highlights the need for this
benchmark to foster future AI advancements.
684909e942e4f9106973f386
Yuki Imajuku
684909e942e4f9106973f387
Kohki Horie
684909e942e4f9106973f388
Yoichi Iwata
684909e942e4f9106973f389
Kensho Aoki
684909e942e4f9106973f38a
Naohiro Takahashi
684909e942e4f9106973f38b
Takuya Akiba
claimed_verified
2025-06-17T07:23:16.335Z
6482810dba6c556892f6f257
https://cdn-avatars.huggingface.co/v1/production/uploads/6482810dba6c556892f6f257/c7-wiVKenXiRtwnRpnjZN.jpeg
Takuya Akiba
user
iwiwi
ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm
  Engineering
How well do AI systems perform in algorithm engineering for hard optimization
problems in domains such as package-delivery routing, crew scheduling, factory
production planning, and power-grid balancing? We introduce ALE-Bench, a new
benchmark for evaluating AI systems on score-based algorithmic programming
contests. Drawing on real tasks from the AtCoder Heuristic Contests, ALE-Bench
presents optimization problems that are computationally hard and admit no known
exact solution. Unlike short-duration, pass/fail coding benchmarks, ALE-Bench
encourages iterative solution refinement over long time horizons. Our software
framework supports interactive agent architectures that leverage test-run
feedback and visualizations. Our evaluation of frontier LLMs revealed that
while they demonstrate high performance on specific problems, a notable gap
remains compared to humans in terms of consistency across problems and
long-horizon problem-solving capabilities. This highlights the need for this
benchmark to foster future AI advancements.
ALE-Bench evaluates AI systems on score-based algorithmic programming contests drawn from AtCoder, focusing on long-term iterative problem-solving in domains like package-delivery routing, crew scheduling, factory production, and power-grid balancing.
algorithm engineering optimization problems ALE-Bench AtCoder Heuristic Contests interactive agent architectures long-horizon problem-solving frontier LLMs