CodeContests+: High-Quality Test Case Generation for Competitive
  Programming
Competitive programming, due to its high reasoning difficulty and precise
correctness feedback, has become a key task for both training and evaluating
the reasoning capabilities of large language models (LLMs). However, while a
large amount of public problem data, such as problem statements and solutions,
is available, the test cases of these problems are often difficult to obtain.
Therefore, test case generation is a necessary task for building large-scale
datasets, and the quality of the test cases directly determines the accuracy of
the evaluation. In this paper, we introduce an LLM-based agent system that
creates high-quality test cases for competitive programming problems. We apply
this system to the CodeContests dataset and propose a new version with improved
test cases, named CodeContests+. We evaluated the quality of test cases in
CodeContestsPlus. First, we used 1.72 million submissions with pass/fail labels
to examine the accuracy of these test cases in evaluation. The results
indicated that CodeContests+ achieves significantly higher accuracy than
CodeContests, particularly with a notably higher True Positive Rate (TPR).
Subsequently, our experiments in LLM Reinforcement Learning (RL) further
confirmed that improvements in test case quality yield considerable advantages
for RL.
68467e323ec10bdd8ab4db44
Zihan Wang
claimed_verified
2025-06-09T10:11:13.717Z
68345ea8beb0d467e37cd421
/avatars/70ddbf00db1c517d61af3a3d283edf42.svg
Zihan Wang
user
zhwang01
68467e323ec10bdd8ab4db45
Siyao Liu
68467e323ec10bdd8ab4db46
Yang Sun
68467e323ec10bdd8ab4db47
Hongyan Li
68467e323ec10bdd8ab4db48
Kai Shen
CodeContests+: High-Quality Test Case Generation for Competitive
  Programming
Competitive programming, due to its high reasoning difficulty and precise
correctness feedback, has become a key task for both training and evaluating
the reasoning capabilities of large language models (LLMs). However, while a
large amount of public problem data, such as problem statements and solutions,
is available, the test cases of these problems are often difficult to obtain.
Therefore, test case generation is a necessary task for building large-scale
datasets, and the quality of the test cases directly determines the accuracy of
the evaluation. In this paper, we introduce an LLM-based agent system that
creates high-quality test cases for competitive programming problems. We apply
this system to the CodeContests dataset and propose a new version with improved
test cases, named CodeContests+. We evaluated the quality of test cases in
CodeContestsPlus. First, we used 1.72 million submissions with pass/fail labels
to examine the accuracy of these test cases in evaluation. The results
indicated that CodeContests+ achieves significantly higher accuracy than
CodeContests, particularly with a notably higher True Positive Rate (TPR).
Subsequently, our experiments in LLM Reinforcement Learning (RL) further
confirmed that improvements in test case quality yield considerable advantages
for RL.
An LLM-based system generates high-quality test cases for competitive programming problems, enhancing the accuracy of model evaluation and RL performance.
LLM-based agent system test case generation CodeContests CodeContests+ True Positive Rate (TPR) LLM Reinforcement Learning (RL)