ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm
  Engineering
How well do AI systems perform in algorithm engineering for hard optimization
problems in domains such as package-delivery routing, crew scheduling, factory
production planning, and power-grid balancing? We introduce ALE-Bench, a new
benchmark for evaluating AI systems on score-based algorithmic programming
contests. Drawing on real tasks from the AtCoder Heuristic Contests, ALE-Bench
presents optimization problems that are computationally hard and admit no known
exact solution. Unlike short-duration, pass/fail coding benchmarks, ALE-Bench
encourages iterative solution refinement over long time horizons. Our software
framework supports interactive agent architectures that leverage test-run
feedback and visualizations. Our evaluation of frontier LLMs revealed that
while they demonstrate high performance on specific problems, a notable gap
remains compared to humans in terms of consistency across problems and
long-horizon problem-solving capabilities. This highlights the need for this
benchmark to foster future AI advancements.
684909e942e4f9106973f386
Yuki Imajuku
claimed_verified
2025-07-01T10:14:50.950Z
65d86a59bdb95b4bbc744e10
https://cdn-avatars.huggingface.co/v1/production/uploads/65d86a59bdb95b4bbc744e10/TUYziiCbQr8JN0LZSxoTY.jpeg
Yuki Imajuku
user
yuki-imajuku
684909e942e4f9106973f387
Kohki Horie
684909e942e4f9106973f388
Yoichi Iwata
684909e942e4f9106973f389
Kensho Aoki
684909e942e4f9106973f38a
Naohiro Takahashi
684909e942e4f9106973f38b
Takuya Akiba
claimed_verified
2025-06-17T07:23:16.335Z
6482810dba6c556892f6f257
https://cdn-avatars.huggingface.co/v1/production/uploads/6482810dba6c556892f6f257/c7-wiVKenXiRtwnRpnjZN.jpeg
Takuya Akiba
user
iwiwi
ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm
  Engineering
How well do AI systems perform in algorithm engineering for hard optimization
problems in domains such as package-delivery routing, crew scheduling, factory
production planning, and power-grid balancing? We introduce ALE-Bench, a new
benchmark for evaluating AI systems on score-based algorithmic programming
contests. Drawing on real tasks from the AtCoder Heuristic Contests, ALE-Bench
presents optimization problems that are computationally hard and admit no known
exact solution. Unlike short-duration, pass/fail coding benchmarks, ALE-Bench
encourages iterative solution refinement over long time horizons. Our software
framework supports interactive agent architectures that leverage test-run
feedback and visualizations. Our evaluation of frontier LLMs revealed that
while they demonstrate high performance on specific problems, a notable gap
remains compared to humans in terms of consistency across problems and
long-horizon problem-solving capabilities. This highlights the need for this
benchmark to foster future AI advancements.
ALE-Bench evaluates AI systems on score-based algorithmic programming contests drawn from AtCoder, focusing on long-term iterative problem-solving in domains like package-delivery routing, crew scheduling, factory production, and power-grid balancing.
algorithm engineering optimization problems ALE-Bench AtCoder Heuristic Contests interactive agent architectures long-horizon problem-solving frontier LLMs