GenPRM: Scaling Test-Time Compute of Process Reward Models via
  Generative Reasoning
Recent advancements in Large Language Models (LLMs) have shown that it is
promising to utilize Process Reward Models (PRMs) as verifiers to enhance the
performance of LLMs. However, current PRMs face three key challenges: (1)
limited process supervision and generalization capabilities, (2) dependence on
scalar value prediction without leveraging the generative abilities of LLMs,
and (3) inability to scale the test-time compute of PRMs. In this work, we
introduce GenPRM, a generative process reward model that performs explicit
Chain-of-Thought (CoT) reasoning with code verification before providing
judgment for each reasoning step. To obtain high-quality process supervision
labels and rationale data, we propose Relative Progress Estimation (RPE) and a
rationale synthesis framework that incorporates code verification. Experimental
results on ProcessBench and several mathematical reasoning tasks show that
GenPRM significantly outperforms prior PRMs with only 23K training data from
MATH dataset. Through test-time scaling, a 1.5B GenPRM outperforms GPT-4o, and
a 7B GenPRM surpasses Qwen2.5-Math-PRM-72B on ProcessBench. Additionally,
GenPRM demonstrates strong abilities to serve as a critic model for policy
model refinement. This work establishes a new paradigm for process supervision
that bridges the gap between PRMs and critic models in LLMs. Our code, model,
and data will be available in https://ryanliu112.github.io/GenPRM.
67ef62342a18e60aeee0ea02
Jian Zhao
67ef62342a18e60aeee0ea03
Runze Liu
claimed_verified
2025-04-04T07:08:09.923Z
667187ba9ab144eb3ac43a1b
/avatars/db5558aa1c5160b9aee8b58573271959.svg
Runze Liu
user
RyanLiu112
67ef62342a18e60aeee0ea04
Kaiyan Zhang
admin_assigned
2025-04-04T07:25:22.709Z
60bc94cd85a3ab33829b6211
/avatars/b57d36c7577fbbb42ea5b963eef4144a.svg
Kaiyan Zhang
user
iseesaw
67ef62342a18e60aeee0ea05
Zhimu Zhou
67ef62342a18e60aeee0ea06
Junqi Gao
admin_assigned
2025-04-04T07:25:38.624Z
67ab05fe4c6ca2d5db4c0c52
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/QpGUNDkeuKjX71s2GXlXF.png
Junqi Gao
user
ChetKao
67ef62342a18e60aeee0ea07
Dong Li
67ef62342a18e60aeee0ea08
Jiafei Lyu
admin_assigned
2025-04-04T07:25:45.394Z
6562db314e8918182da42706
/avatars/b113bbbb496bf4dac254f0e840f08e10.svg
Jiafei Lyu
user
dmux
67ef62342a18e60aeee0ea09
Zhouyi Qian
admin_assigned
2025-04-04T07:26:00.763Z
65b34c5785b6c2144807db37
/avatars/4c1cb03cda250d4ec760ebf7815a3bce.svg
Qianzhouyi
user
Saputello
67ef62342a18e60aeee0ea0a
Biqing Qi
admin_assigned
2025-04-04T07:26:06.517Z
645d9c3058f9ee315148116d
https://cdn-avatars.huggingface.co/v1/production/uploads/645d9c3058f9ee315148116d/uBoAWgrF2Di4WcXVGW9fP.jpeg
Biqing Qi
user
jackqi7
67ef62342a18e60aeee0ea0b
Xiu Li
67ef62342a18e60aeee0ea0c
Bowen Zhou
admin_assigned
2025-04-04T07:26:14.312Z
669f614b59adf5b56e05bce3
/avatars/ffd4189efbceb0e63a03db273065a44b.svg
BowenZhou
user
bowenZhou
GenPRM: Scaling Test-Time Compute of Process Reward Models via
  Generative Reasoning
Recent advancements in Large Language Models (LLMs) have shown that it is
promising to utilize Process Reward Models (PRMs) as verifiers to enhance the
performance of LLMs. However, current PRMs face three key challenges: (1)
limited process supervision and generalization capabilities, (2) dependence on
scalar value prediction without leveraging the generative abilities of LLMs,
and (3) inability to scale the test-time compute of PRMs. In this work, we
introduce GenPRM, a generative process reward model that performs explicit
Chain-of-Thought (CoT) reasoning with code verification before providing
judgment for each reasoning step. To obtain high-quality process supervision
labels and rationale data, we propose Relative Progress Estimation (RPE) and a
rationale synthesis framework that incorporates code verification. Experimental
results on ProcessBench and several mathematical reasoning tasks show that
GenPRM significantly outperforms prior PRMs with only 23K training data from
MATH dataset. Through test-time scaling, a 1.5B GenPRM outperforms GPT-4o, and
a 7B GenPRM surpasses Qwen2.5-Math-PRM-72B on ProcessBench. Additionally,
GenPRM demonstrates strong abilities to serve as a critic model for policy
model refinement. This work establishes a new paradigm for process supervision
that bridges the gap between PRMs and critic models in LLMs. Our code, model,
and data will be available in https://ryanliu112.github.io/GenPRM.
GenPRM, a generative process reward model with Chain-of-Thought reasoning and code verification, outperforms existing PRMs and critic models in process supervision tasks with less data and computational resources.
Large Language Models Process Reward Models Chain-of-Thought Relative Progress Estimation rationale synthesis GenPRM ProcessBench mathematical reasoning tasks critic model policy model refinement