AetherCode: Evaluating LLMs' Ability to Win In Premier Programming
  Competitions
Competitive programming has emerged as a critical benchmark for evaluating
the reasoning and coding capabilities of Large Language Models (LLMs). Despite
impressive progress on existing benchmarks, we argue that current evaluations
overstate model proficiency, masking a substantial gap between LLMs and elite
human programmers. This gap arises from two key limitations: insufficient
difficulty and scope of benchmark problems, and evaluation bias from
low-quality test cases. To address these shortcomings, we present AetherCode, a
new benchmark that draws problems from premier programming competitions such as
IOI and ICPC, offering broader coverage and higher difficulty. AetherCode
further incorporates comprehensive, expert-validated test suites built through
a hybrid of automated generation and human curation, ensuring rigorous and
reliable assessment. By combining challenging problem design with robust
evaluation, AetherCode provides a more faithful measure of LLM capabilities and
sets a new standard for future research in code reasoning.
68abd69686b21a0e2e358a9a
Zihan Wang
claimed_verified
2025-08-25T08:18:53.636Z
68345ea8beb0d467e37cd421
/avatars/70ddbf00db1c517d61af3a3d283edf42.svg
Zihan Wang
user
zhwang01
68abd69686b21a0e2e358a9b
Jiaze Chen
68abd69686b21a0e2e358a9c
Zhicheng Liu
68abd69686b21a0e2e358a9d
Markus Mak
68abd69686b21a0e2e358a9e
Yidi Du
68abd69686b21a0e2e358a9f
Geonsik Moon
68abd69686b21a0e2e358aa0
Luoqi Xu
68abd69686b21a0e2e358aa1
Aaron Tua
68abd69686b21a0e2e358aa2
Kunshuo Peng
68abd69686b21a0e2e358aa3
Jiayi Lu
68abd69686b21a0e2e358aa4
Mingfei Xia
68abd69686b21a0e2e358aa5
Boqian Zou
68abd69686b21a0e2e358aa6
Chenyang Ran
68abd69686b21a0e2e358aa7
Guang Tian
68abd69686b21a0e2e358aa8
Shoutai Zhu
68abd69686b21a0e2e358aa9
Yeheng Duan
68abd69686b21a0e2e358aaa
Zhenghui Kang
68abd69686b21a0e2e358aab
Zhenxing Lin
68abd69686b21a0e2e358aac
Shangshu Li
68abd69686b21a0e2e358aad
Qiang Luo
68abd69686b21a0e2e358aae
Qingshen Long
68abd69686b21a0e2e358aaf
Zhiyong Chen
68abd69686b21a0e2e358ab0
Yihan Xiao
68abd69686b21a0e2e358ab1
Yurong Wu
68abd69686b21a0e2e358ab2
Daoguang Zan
68abd69686b21a0e2e358ab3
Yuyi Fu
68abd69686b21a0e2e358ab4
Mingxuan Wang
68abd69686b21a0e2e358ab5
Ming Ding
AetherCode: Evaluating LLMs' Ability to Win In Premier Programming
  Competitions
Competitive programming has emerged as a critical benchmark for evaluating
the reasoning and coding capabilities of Large Language Models (LLMs). Despite
impressive progress on existing benchmarks, we argue that current evaluations
overstate model proficiency, masking a substantial gap between LLMs and elite
human programmers. This gap arises from two key limitations: insufficient
difficulty and scope of benchmark problems, and evaluation bias from
low-quality test cases. To address these shortcomings, we present AetherCode, a
new benchmark that draws problems from premier programming competitions such as
IOI and ICPC, offering broader coverage and higher difficulty. AetherCode
further incorporates comprehensive, expert-validated test suites built through
a hybrid of automated generation and human curation, ensuring rigorous and
reliable assessment. By combining challenging problem design with robust
evaluation, AetherCode provides a more faithful measure of LLM capabilities and
sets a new standard for future research in code reasoning.
AetherCode is a new benchmark for evaluating Large Language Models in competitive programming, offering more challenging and expert-validated test cases than existing benchmarks.
Large Language Models competitive programming benchmark reasoning coding capabilities evaluation bias IOI ICPC test suites automated generation human curation code reasoning