HardTests: Synthesizing High-Quality Test Cases for LLM Coding
Verifiers play a crucial role in large language model (LLM) reasoning, needed
by post-training techniques such as reinforcement learning. However, reliable
verifiers are hard to get for difficult coding problems, because a
well-disguised wrong solution may only be detected by carefully human-written
edge cases that are difficult to synthesize. To address this issue, we propose
HARDTESTGEN, a pipeline for high-quality test synthesis using LLMs. With this
pipeline, we curate a comprehensive competitive programming dataset HARDTESTS
with 47k problems and synthetic high-quality tests. Compared with existing
tests, HARDTESTGEN tests demonstrate precision that is 11.3 percentage points
higher and recall that is 17.5 percentage points higher when evaluating
LLM-generated code. For harder problems, the improvement in precision can be as
large as 40 points. HARDTESTS also proves to be more effective for model
training, measured by downstream code generation performance. We will
open-source our dataset and synthesis pipeline at
https://leililab.github.io/HardTests/.
683d2cee5bdbb3803e42bc8a
Zhongmou He
683d2cee5bdbb3803e42bc8b
Yee Man Choi
683d2cee5bdbb3803e42bc8c
Kexun Zhang
683d2cee5bdbb3803e42bc8d
Jiabao Ji
683d2cee5bdbb3803e42bc8e
Junting Zhou
claimed_verified
2025-06-02T07:40:34.926Z
65a374a59acab1998092a9bc
https://cdn-avatars.huggingface.co/v1/production/uploads/65a374a59acab1998092a9bc/M3s_7bSf9G-6b9nLg7N3Z.jpeg
Antonio
user
JuntingZhou
683d2cee5bdbb3803e42bc8f
Dejia Xu
683d2cee5bdbb3803e42bc90
Ivan Bercovich
claimed_verified
2025-06-10T09:29:32.638Z
638a9361f560ea995b58c7e7
https://cdn-avatars.huggingface.co/v1/production/uploads/638a9361f560ea995b58c7e7/pn40t46XsrB-6pKEmqAv-.jpeg
Ivan Bercovich
user
ibercovich
683d2cee5bdbb3803e42bc91
Aidan Zhang
683d2cee5bdbb3803e42bc92
Lei Li
HardTests: Synthesizing High-Quality Test Cases for LLM Coding
Verifiers play a crucial role in large language model (LLM) reasoning, needed
by post-training techniques such as reinforcement learning. However, reliable
verifiers are hard to get for difficult coding problems, because a
well-disguised wrong solution may only be detected by carefully human-written
edge cases that are difficult to synthesize. To address this issue, we propose
HARDTESTGEN, a pipeline for high-quality test synthesis using LLMs. With this
pipeline, we curate a comprehensive competitive programming dataset HARDTESTS
with 47k problems and synthetic high-quality tests. Compared with existing
tests, HARDTESTGEN tests demonstrate precision that is 11.3 percentage points
higher and recall that is 17.5 percentage points higher when evaluating
LLM-generated code. For harder problems, the improvement in precision can be as
large as 40 points. HARDTESTS also proves to be more effective for model
training, measured by downstream code generation performance. We will
open-source our dataset and synthesis pipeline at
https://leililab.github.io/HardTests/.
HARDTESTGEN creates a large, high-quality competitive programming dataset to enhance the precision and recall of verifiers in evaluating LLM-generated code.
LLM reasoning reinforcement learning verifiers test synthesis LLMs competitive programming synthetic tests precision recall code generation performance