LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive
  Programming?
Recent reports claim that large language models (LLMs) now outperform elite
humans in competitive programming. Drawing on knowledge from a group of
medalists in international algorithmic contests, we revisit this claim,
examining how LLMs differ from human experts and where limitations still
remain. We introduce LiveCodeBench Pro, a benchmark composed of problems from
Codeforces, ICPC, and IOI that are continuously updated to reduce the
likelihood of data contamination. A team of Olympiad medalists annotates every
problem for algorithmic categories and conducts a line-by-line analysis of
failed model-generated submissions. Using this new data and benchmark, we find
that frontier models still have significant limitations: without external
tools, the best model achieves only 53% pass@1 on medium-difficulty problems
and 0% on hard problems, domains where expert humans still excel. We also find
that LLMs succeed at implementation-heavy problems but struggle with nuanced
algorithmic reasoning and complex case analysis, often generating confidently
incorrect justifications. High performance appears largely driven by
implementation precision and tool augmentation, not superior reasoning.
LiveCodeBench Pro thus highlights the significant gap to human grandmaster
levels, while offering fine-grained diagnostics to steer future improvements in
code-centric LLM reasoning.
684fae8d60b4a34dbe007acd
Zihan Zheng
684fae8d60b4a34dbe007ace
Zerui Cheng
claimed_verified
2025-06-16T15:50:50.311Z
67d8e5f041fc44bb16e5468c
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/jUCe0rf7Z-cF51OC5Iczn.png
Zerui Cheng
user
ZeruiCheng
684fae8d60b4a34dbe007acf
Zeyu Shen
claimed_verified
2025-06-16T15:50:52.696Z
672857ff73515943ca416973
/avatars/5f630a6b867c46fdd0fc1996186fef4d.svg
Zeyu Shen
user
zeyush
684fae8d60b4a34dbe007ad0
Shang Zhou
684fae8d60b4a34dbe007ad1
Kaiyuan Liu
684fae8d60b4a34dbe007ad2
Hansen He
684fae8d60b4a34dbe007ad3
Dongruixuan Li
684fae8d60b4a34dbe007ad4
Stanley Wei
claimed_verified
2025-06-20T12:20:15.913Z
685498d378c471e64b7e7a67
/avatars/746c2c24e52820e1667395e0a2870f31.svg
Stanley Wei
user
stanleyrwei
684fae8d60b4a34dbe007ad5
Hangyi Hao
claimed_verified
2025-06-25T08:09:36.493Z
68520dce22479d7f673f2e0f
https://cdn-avatars.huggingface.co/v1/production/uploads/68520dce22479d7f673f2e0f/PsckFsdtTbEj_vnwNItYN.jpeg
Hangyi Hao
user
HathawayHao
684fae8d60b4a34dbe007ad6
Jianzhu Yao
684fae8d60b4a34dbe007ad7
Peiyao Sheng
admin_assigned
2025-06-16T13:00:36.911Z
681e8c219afd4ecd3e2c19f8
/avatars/2ddd4e0b5de7ffa060141243cbc2410a.svg
Peiyao Sheng
user
peiyao-sentient
684fae8d60b4a34dbe007ad8
Zixuan Wang
684fae8d60b4a34dbe007ad9
Wenhao Chai
claimed_verified
2025-06-16T15:50:54.781Z
637c7503fe115289cfecbe6b
https://cdn-avatars.huggingface.co/v1/production/uploads/1676361945047-637c7503fe115289cfecbe6b.jpeg
Wenhao Chai
user
wchai
684fae8d60b4a34dbe007ada
Aleksandra Korolova
admin_assigned
2025-06-16T13:00:20.576Z
66aff5f4f0bb52918956631a
/avatars/12dfc9b534e2dc5d0e98c3723af2ca97.svg
Aleksandra Korolova
user
korolova
684fae8d60b4a34dbe007adb
Peter Henderson
claimed_verified
2025-06-26T09:33:09.043Z
617aafa1ff3db6021d069787
https://cdn-avatars.huggingface.co/v1/production/uploads/1644543485859-617aafa1ff3db6021d069787.jpeg
Peter Henderson
user
breakend
684fae8d60b4a34dbe007adc
Sanjeev Arora
684fae8d60b4a34dbe007add
Pramod Viswanath
684fae8d60b4a34dbe007ade
Jingbo Shang
admin_assigned
2025-06-16T13:00:07.437Z
660655119e3555d648f6c6b5
/avatars/ae1e2c97a08be39b77a9f1a5c2a718ef.svg
Jingbo Shang
user
shangjingbo
684fae8d60b4a34dbe007adf
Saining Xie
admin_assigned
2025-06-16T13:00:01.691Z
6596422646624a86ff3b3bda
/avatars/216e12b77e45ac5f1fa20932f5745411.svg
Saining Xie
user
sainx
LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive
  Programming?
Recent reports claim that large language models (LLMs) now outperform elite
humans in competitive programming. Drawing on knowledge from a group of
medalists in international algorithmic contests, we revisit this claim,
examining how LLMs differ from human experts and where limitations still
remain. We introduce LiveCodeBench Pro, a benchmark composed of problems from
Codeforces, ICPC, and IOI that are continuously updated to reduce the
likelihood of data contamination. A team of Olympiad medalists annotates every
problem for algorithmic categories and conducts a line-by-line analysis of
failed model-generated submissions. Using this new data and benchmark, we find
that frontier models still have significant limitations: without external
tools, the best model achieves only 53% pass@1 on medium-difficulty problems
and 0% on hard problems, domains where expert humans still excel. We also find
that LLMs succeed at implementation-heavy problems but struggle with nuanced
algorithmic reasoning and complex case analysis, often generating confidently
incorrect justifications. High performance appears largely driven by
implementation precision and tool augmentation, not superior reasoning.
LiveCodeBench Pro thus highlights the significant gap to human grandmaster
levels, while offering fine-grained diagnostics to steer future improvements in
code-centric LLM reasoning.
LLMs perform well on implementation-heavy competitive programming problems but struggle with nuanced algorithmic reasoning, as highlighted by LiveCodeBench Pro.
large language models LLMs competitive programming LiveCodeBench Pro Codeforces ICPC IOI algorithmic categories algorithmic reasoning case analysis