SCAN: Self-Denoising Monte Carlo Annotation for Robust Process Reward
  Learning
Process reward models (PRMs) offer fine-grained, step-level evaluations that
facilitate deeper reasoning processes in large language models (LLMs), proving
effective in complex tasks like mathematical reasoning. However, developing
PRMs is challenging due to the high cost and limited scalability of
human-annotated data. Synthetic data from Monte Carlo (MC) estimation is a
promising alternative but suffers from a high noise ratio, which can cause
overfitting and hinder large-scale training. In this work, we conduct a
preliminary study on the noise distribution in synthetic data from MC
estimation, identifying that annotation models tend to both underestimate and
overestimate step correctness due to limitations in their annotation
capabilities. Building on these insights, we propose Self-Denoising Monte Carlo
Annotation (SCAN), an efficient data synthesis and noise-tolerant learning
framework. Our key findings indicate that: (1) Even lightweight models (e.g.,
1.5B parameters) can produce high-quality annotations through a self-denoising
strategy, enabling PRMs to achieve superior performance with only 6% the
inference cost required by vanilla MC estimation. (2) With our robust learning
strategy, PRMs can effectively learn from this weak supervision, achieving a
39.2 F1 score improvement (from 19.9 to 59.1) in ProcessBench. Despite using
only a compact synthetic dataset, our models surpass strong baselines,
including those trained on large-scale human-annotated datasets such as
PRM800K. Furthermore, performance continues to improve as we scale up the
synthetic data, highlighting the potential of SCAN for scalable,
cost-efficient, and robust PRM training.
68d237f71ca7156988a8ee42
Yuyang Ding
claimed_verified
2025-09-23T10:05:04.637Z
626cf0f65651e31a7a2b9779
https://cdn-avatars.huggingface.co/v1/production/uploads/626cf0f65651e31a7a2b9779/TAE--QIKo1vz4Rb8aearl.jpeg
Ding
user
dyyyyyyyy
68d237f71ca7156988a8ee43
Xinyu Shi
admin_assigned
2025-09-23T14:28:19.824Z
6380ff85f496d57325c2ac2b
/avatars/0b9eb9c8a96d5ffd905f9ddeee78e6f3.svg
Xinyu Shi
user
XinyuShi
68d237f71ca7156988a8ee44
Juntao Li
68d237f71ca7156988a8ee45
Xiaobo Liang
admin_assigned
2025-09-23T14:28:34.022Z
64116770230ce11b1be7f3b2
/avatars/d554406a121666c3e16bc3321415ccb1.svg
Xiaobo Liang
user
dropreg
68d237f71ca7156988a8ee46
Zhaopeng Tu
admin_assigned
2025-09-23T14:28:41.973Z
67485743561b1e6f9579389f
/avatars/8a4cc63bd7be388010bc329bb74582a1.svg
Zhaopeng Tu
user
zptu
68d237f71ca7156988a8ee47
Min Zhang
SCAN: Self-Denoising Monte Carlo Annotation for Robust Process Reward
  Learning
Process reward models (PRMs) offer fine-grained, step-level evaluations that
facilitate deeper reasoning processes in large language models (LLMs), proving
effective in complex tasks like mathematical reasoning. However, developing
PRMs is challenging due to the high cost and limited scalability of
human-annotated data. Synthetic data from Monte Carlo (MC) estimation is a
promising alternative but suffers from a high noise ratio, which can cause
overfitting and hinder large-scale training. In this work, we conduct a
preliminary study on the noise distribution in synthetic data from MC
estimation, identifying that annotation models tend to both underestimate and
overestimate step correctness due to limitations in their annotation
capabilities. Building on these insights, we propose Self-Denoising Monte Carlo
Annotation (SCAN), an efficient data synthesis and noise-tolerant learning
framework. Our key findings indicate that: (1) Even lightweight models (e.g.,
1.5B parameters) can produce high-quality annotations through a self-denoising
strategy, enabling PRMs to achieve superior performance with only 6% the
inference cost required by vanilla MC estimation. (2) With our robust learning
strategy, PRMs can effectively learn from this weak supervision, achieving a
39.2 F1 score improvement (from 19.9 to 59.1) in ProcessBench. Despite using
only a compact synthetic dataset, our models surpass strong baselines,
including those trained on large-scale human-annotated datasets such as
PRM800K. Furthermore, performance continues to improve as we scale up the
synthetic data, highlighting the potential of SCAN for scalable,
cost-efficient, and robust PRM training.
SCAN, a self-denoising Monte Carlo framework, improves PRM performance with synthetic data, achieving high F1 scores and surpassing human-annotated baselines.
Process reward models Monte Carlo estimation self-denoising noise distribution annotation models ProcessBench F1 score PRM800K