Hard2Verify: A Step-Level Verification Benchmark for Open-Ended Frontier
  Math
Large language model (LLM)-based reasoning systems have recently achieved
gold medal-level performance in the IMO 2025 competition, writing mathematical
proofs where, to receive full credit, each step must be not only correct but
also sufficiently supported. To train LLM-based reasoners in such challenging,
open-ended settings, strong verifiers capable of catching step-level mistakes
are necessary prerequisites. We introduce Hard2Verify, a human-annotated,
step-level verification benchmark produced with over 500 hours of human labor.
Hard2Verify is designed to rigorously assess step-level verifiers at the
frontier: Verifiers must provide step-level annotations or identify the first
error in responses generated by frontier LLMs for very recent, challenging, and
open-ended math questions. We evaluate 29 generative critics and process reward
models, demonstrating that, beyond a few standouts, open-source verifiers lag
closed source models. We subsequently analyze what drives poor performance in
step-level verification, the impacts of scaling verifier compute, as well as
fundamental questions such as self-verification and verification-generation
dynamics.
68f0495d36f8b025381e18b7
Shrey Pandit
claimed_verified
2025-10-17T04:14:09.981Z
648749094dea003c6dae810f
https://cdn-avatars.huggingface.co/v1/production/uploads/648749094dea003c6dae810f/gHUHSBt1zrt8wjO1YwTNu.jpeg
Shrey Pandit
user
SP2001
68f0495d36f8b025381e18b8
Austin Xu
claimed_verified
2025-10-18T04:50:04.953Z
6668e86dc4ef4175fb18d250
/avatars/d34925685799a1218017f85ce4d44e6e.svg
Austin Xu
user
austinxu87
68f0495d36f8b025381e18b9
Xuan-Phi Nguyen
68f0495d36f8b025381e18ba
Yifei Ming
68f0495d36f8b025381e18bb
Caiming Xiong
68f0495d36f8b025381e18bc
Shafiq Joty
Hard2Verify: A Step-Level Verification Benchmark for Open-Ended Frontier
  Math
Large language model (LLM)-based reasoning systems have recently achieved
gold medal-level performance in the IMO 2025 competition, writing mathematical
proofs where, to receive full credit, each step must be not only correct but
also sufficiently supported. To train LLM-based reasoners in such challenging,
open-ended settings, strong verifiers capable of catching step-level mistakes
are necessary prerequisites. We introduce Hard2Verify, a human-annotated,
step-level verification benchmark produced with over 500 hours of human labor.
Hard2Verify is designed to rigorously assess step-level verifiers at the
frontier: Verifiers must provide step-level annotations or identify the first
error in responses generated by frontier LLMs for very recent, challenging, and
open-ended math questions. We evaluate 29 generative critics and process reward
models, demonstrating that, beyond a few standouts, open-source verifiers lag
closed source models. We subsequently analyze what drives poor performance in
step-level verification, the impacts of scaling verifier compute, as well as
fundamental questions such as self-verification and verification-generation
dynamics.
Hard2Verify, a human-annotated benchmark, evaluates step-level verifiers for LLM-based mathematical reasoning systems, highlighting the challenges and performance gaps between open-source and closed-source models.
Large language model LLM-based reasoning systems IMO 2025 mathematical proofs step-level verification Hard2Verify generative critics process reward models self-verification verification-generation dynamics