Uncertainty-Based Methods for Automated Process Reward Data Construction
  and Output Aggregation in Mathematical Reasoning
Large language models have demonstrated remarkable capabilities in complex
mathematical reasoning tasks, but they inevitably generate errors throughout
multi-step solutions. Process-level Reward Models (PRMs) have shown great
promise by providing supervision and evaluation at each intermediate step,
thereby effectively improving the models' reasoning abilities. However,
training effective PRMs requires high-quality process reward data, yet existing
methods for constructing such data are often labour-intensive or inefficient.
In this paper, we propose an uncertainty-driven framework for automated process
reward data construction, encompassing both data generation and annotation
processes for PRMs. Additionally, we identify the limitations of both majority
vote and PRMs, and introduce two generic uncertainty-aware output aggregation
methods: Hybrid Majority Reward Vote and Weighted Reward Frequency Vote, which
combine the strengths of majority vote with PRMs. Extensive experiments on
ProcessBench, MATH, and GSMPlus show the effectiveness and efficiency of the
proposed PRM data construction framework, and demonstrate that the two output
aggregation methods further improve the mathematical reasoning abilities across
diverse PRMs. The code and data will be publicly available at
https://github.com/Jiuzhouh/UnPRM.
6891fbb39373cd501b6a2d94
Jiuzhou Han
6891fbb39373cd501b6a2d95
Wray Buntine
6891fbb39373cd501b6a2d96
Ehsan Shareghi
Uncertainty-Based Methods for Automated Process Reward Data Construction
  and Output Aggregation in Mathematical Reasoning
Large language models have demonstrated remarkable capabilities in complex
mathematical reasoning tasks, but they inevitably generate errors throughout
multi-step solutions. Process-level Reward Models (PRMs) have shown great
promise by providing supervision and evaluation at each intermediate step,
thereby effectively improving the models' reasoning abilities. However,
training effective PRMs requires high-quality process reward data, yet existing
methods for constructing such data are often labour-intensive or inefficient.
In this paper, we propose an uncertainty-driven framework for automated process
reward data construction, encompassing both data generation and annotation
processes for PRMs. Additionally, we identify the limitations of both majority
vote and PRMs, and introduce two generic uncertainty-aware output aggregation
methods: Hybrid Majority Reward Vote and Weighted Reward Frequency Vote, which
combine the strengths of majority vote with PRMs. Extensive experiments on
ProcessBench, MATH, and GSMPlus show the effectiveness and efficiency of the
proposed PRM data construction framework, and demonstrate that the two output
aggregation methods further improve the mathematical reasoning abilities across
diverse PRMs. The code and data will be publicly available at
https://github.com/Jiuzhouh/UnPRM.
An uncertainty-driven framework for automated process reward data construction and aggregation methods improves the effectiveness and efficiency of Process-Level Reward Models in mathematical reasoning tasks.
Process-Level Reward Models PRMs uncertainty-driven framework data generation annotation processes Hybrid Majority Reward Vote Weighted Reward Frequency Vote ProcessBench MATH GSMPlus