The Lessons of Developing Process Reward Models in Mathematical
  Reasoning
Process Reward Models (PRMs) emerge as a promising approach for process
supervision in mathematical reasoning of Large Language Models (LLMs), which
aim to identify and mitigate intermediate errors in the reasoning processes.
However, the development of effective PRMs faces significant challenges,
particularly in data annotation and evaluation methodologies. In this paper,
through extensive experiments, we demonstrate that commonly used Monte Carlo
(MC) estimation-based data synthesis for PRMs typically yields inferior
performance and generalization compared to LLM-as-a-judge and human annotation
methods. MC estimation relies on completion models to evaluate current-step
correctness, leading to inaccurate step verification. Furthermore, we identify
potential biases in conventional Best-of-N (BoN) evaluation strategies for
PRMs: (1) The unreliable policy models generate responses with correct answers
but flawed processes, leading to a misalignment between the evaluation criteria
of BoN and the PRM objectives of process verification. (2) The tolerance of
PRMs of such responses leads to inflated BoN scores. (3) Existing PRMs have a
significant proportion of minimum scores concentrated on the final answer
steps, revealing the shift from process to outcome-based assessment in BoN
Optimized PRMs. To address these challenges, we develop a consensus filtering
mechanism that effectively integrates MC estimation with LLM-as-a-judge and
advocates a more comprehensive evaluation framework that combines
response-level and step-level metrics. Based on the mechanisms, we
significantly improve both model performance and data efficiency in the BoN
evaluation and the step-wise error identification task. Finally, we release a
new state-of-the-art PRM that outperforms existing open-source alternatives and
provides practical guidelines for future research in building process
supervision models.
6785e601117627fe711f8dd8
Zhenru Zhang
admin_assigned
2025-01-14T08:48:52.704Z
64704e973601bb7b06643e98
/avatars/52e51f4d1be6769e4397b8be2799cf32.svg
Zhenru Zhang
user
Zhenru
6785e601117627fe711f8dd9
Chujie Zheng
claimed_verified
2025-01-14T08:28:43.769Z
610b70452719facd4ea85e28
https://cdn-avatars.huggingface.co/v1/production/uploads/610b70452719facd4ea85e28/S7nMy7D0Rxq0VIVblhYDG.jpeg
Chujie Zheng
user
chujiezheng
6785e601117627fe711f8dda
Yangzhen Wu
admin_assigned
2025-01-14T08:49:01.963Z
671f2a2ab02f88b7e2385105
/avatars/a97e2f983c8290b72eeb03e2bdcaf085.svg
Yangzhen Wu
user
wuyangzhen
6785e601117627fe711f8ddb
Beichen Zhang
admin_assigned
2025-01-14T08:49:43.861Z
64b93578ee257c3a4cfceed1
/avatars/e6188562254f75a09b4048b800860016.svg
Beichen Zhang
user
BeichenZhang
6785e601117627fe711f8ddc
Runji Lin
admin_assigned
2025-01-14T08:49:56.685Z
649a52e5de0fb7f3f499e583
/avatars/25f6106fa168ae57ad5cd8ef55c70d31.svg
Runji Lin
user
RunjiLin
6785e601117627fe711f8ddd
Bowen Yu
admin_assigned
2025-01-14T08:50:12.316Z
6438b43ab2ea24b52ebac2b9
/avatars/84133cd719a4b1e2f5c1a74178425f86.svg
Bowen Yu
user
bwy
6785e601117627fe711f8dde
Dayiheng Liu
admin_assigned
2025-01-14T08:50:18.536Z
6434d4989bd5a84b5dd0b0f5
https://cdn-avatars.huggingface.co/v1/production/uploads/6434d4989bd5a84b5dd0b0f5/0Elf9qbfG9Hkgypm9pTGm.jpeg
Dayiheng Liu
user
Losin94
6785e601117627fe711f8ddf
Jingren Zhou
admin_assigned
2025-01-14T08:50:29.301Z
602f88f5e8149a962412a667
/avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
Zhou
user
Jingren
6785e601117627fe711f8de0
Junyang Lin
admin_assigned
2025-01-14T08:50:35.067Z
620760a26e3b7210c2ff1943
https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
Junyang Lin
user
JustinLin610
The Lessons of Developing Process Reward Models in Mathematical
  Reasoning
Process Reward Models (PRMs) emerge as a promising approach for process
supervision in mathematical reasoning of Large Language Models (LLMs), which
aim to identify and mitigate intermediate errors in the reasoning processes.
However, the development of effective PRMs faces significant challenges,
particularly in data annotation and evaluation methodologies. In this paper,
through extensive experiments, we demonstrate that commonly used Monte Carlo
(MC) estimation-based data synthesis for PRMs typically yields inferior
performance and generalization compared to LLM-as-a-judge and human annotation
methods. MC estimation relies on completion models to evaluate current-step
correctness, leading to inaccurate step verification. Furthermore, we identify
potential biases in conventional Best-of-N (BoN) evaluation strategies for
PRMs: (1) The unreliable policy models generate responses with correct answers
but flawed processes, leading to a misalignment between the evaluation criteria
of BoN and the PRM objectives of process verification. (2) The tolerance of
PRMs of such responses leads to inflated BoN scores. (3) Existing PRMs have a
significant proportion of minimum scores concentrated on the final answer
steps, revealing the shift from process to outcome-based assessment in BoN
Optimized PRMs. To address these challenges, we develop a consensus filtering
mechanism that effectively integrates MC estimation with LLM-as-a-judge and
advocates a more comprehensive evaluation framework that combines
response-level and step-level metrics. Based on the mechanisms, we
significantly improve both model performance and data efficiency in the BoN
evaluation and the step-wise error identification task. Finally, we release a
new state-of-the-art PRM that outperforms existing open-source alternatives and
provides practical guidelines for future research in building process
supervision models.
Process Reward Models improve performance in mathematical reasoning by integrating Monte Carlo estimation with LLM-as-a-judge, addressing biases in Best-of-N evaluation and step-wise error identification.
Process Reward Models Monte Carlo estimation LLM-as-a-judge Best-of-N evaluation process verification step-wise error identification