Challenging the Boundaries of Reasoning: An Olympiad-Level Math
  Benchmark for Large Language Models
In recent years, the rapid development of large reasoning models has resulted
in the saturation of existing benchmarks for evaluating mathematical reasoning,
highlighting the urgent need for more challenging and rigorous evaluation
frameworks. To address this gap, we introduce OlymMATH, a novel Olympiad-level
mathematical benchmark, designed to rigorously test the complex reasoning
capabilities of LLMs. OlymMATH features 200 meticulously curated problems, each
manually verified and available in parallel English and Chinese versions. The
problems are systematically organized into two distinct difficulty tiers: (1)
AIME-level problems (easy) that establish a baseline for mathematical reasoning
assessment, and (2) significantly more challenging problems (hard) designed to
push the boundaries of current state-of-the-art models. In our benchmark, these
problems span four core mathematical fields, each including a verifiable
numerical solution to enable objective, rule-based evaluation. Empirical
results underscore the significant challenge presented by OlymMATH, with
state-of-the-art models including DeepSeek-R1 and OpenAI's o3-mini
demonstrating notably limited accuracy on the hard subset. Furthermore, the
benchmark facilitates comprehensive bilingual assessment of mathematical
reasoning abilities-a critical dimension that remains largely unaddressed in
mainstream mathematical reasoning benchmarks. We release the OlymMATH benchmark
at the STILL project: https://github.com/RUCAIBox/Slow_Thinking_with_LLMs.
67e5f4ad147ee85622ad0df1
Haoxiang Sun
admin_assigned
2025-03-28T08:46:46.418Z
65df408822d66a997b4d5f6e
https://cdn-avatars.huggingface.co/v1/production/uploads/65df408822d66a997b4d5f6e/poROuCSvB39NZSiLzxLZf.jpeg
Haoxiang Sun
user
CoderBak
67e5f4ad147ee85622ad0df2
Yingqian Min
claimed_verified
2025-03-28T08:38:03.334Z
6703ac76ea890f0ca5b225eb
/avatars/5f56c49a1940143d47dd484782a4abbf.svg
Yingqian Min
user
EliverQ
67e5f4ad147ee85622ad0df3
Zhipeng Chen
claimed_verified
2025-03-28T08:38:01.177Z
629b765ce1af194c641fcbc6
/avatars/7c53a4c2a1e528c19641a2b601731754.svg
Zhipeng Chen
user
TimothyCzp
67e5f4ad147ee85622ad0df4
Wayne Xin Zhao
67e5f4ad147ee85622ad0df5
Zheng Liu
67e5f4ad147ee85622ad0df6
Zhongyuan Wang
67e5f4ad147ee85622ad0df7
Lei Fang
67e5f4ad147ee85622ad0df8
Ji-Rong Wen
admin_assigned
2025-03-28T08:47:20.525Z
64b8c89052b7353d8c6a1013
/avatars/cd59fffe81f6b07b4519540b8ff3d95f.svg
Ji-Rong Wen
user
jrwen
Challenging the Boundaries of Reasoning: An Olympiad-Level Math
  Benchmark for Large Language Models
In recent years, the rapid development of large reasoning models has resulted
in the saturation of existing benchmarks for evaluating mathematical reasoning,
highlighting the urgent need for more challenging and rigorous evaluation
frameworks. To address this gap, we introduce OlymMATH, a novel Olympiad-level
mathematical benchmark, designed to rigorously test the complex reasoning
capabilities of LLMs. OlymMATH features 200 meticulously curated problems, each
manually verified and available in parallel English and Chinese versions. The
problems are systematically organized into two distinct difficulty tiers: (1)
AIME-level problems (easy) that establish a baseline for mathematical reasoning
assessment, and (2) significantly more challenging problems (hard) designed to
push the boundaries of current state-of-the-art models. In our benchmark, these
problems span four core mathematical fields, each including a verifiable
numerical solution to enable objective, rule-based evaluation. Empirical
results underscore the significant challenge presented by OlymMATH, with
state-of-the-art models including DeepSeek-R1 and OpenAI's o3-mini
demonstrating notably limited accuracy on the hard subset. Furthermore, the
benchmark facilitates comprehensive bilingual assessment of mathematical
reasoning abilities-a critical dimension that remains largely unaddressed in
mainstream mathematical reasoning benchmarks. We release the OlymMATH benchmark
at the STILL project: https://github.com/RUCAIBox/Slow_Thinking_with_LLMs.
OlymMATH is a new benchmark for evaluating the complex reasoning capabilities of LLMs using challenging Olympiad-level problems in parallel English and Chinese.
LLMs DeepSeek-R1 o3-mini bilingual assessment mathematical reasoning benchmark AIME-level problems