GroundedPRM: Tree-Guided and Fidelity-Aware Process Reward Modeling for
  Step-Level Reasoning
Process Reward Models (PRMs) aim to improve multi-step reasoning in Large
Language Models (LLMs) by supervising intermediate steps and identifying
errors. However, building effective PRMs remains challenging due to the lack of
scalable, high-quality annotations. Existing approaches rely on costly human
labeling, LLM-based self-evaluation that is prone to hallucination, or Monte
Carlo (MC) estimation, which infers step quality solely from rollout outcomes
and often introduces noisy, misaligned supervision due to credit
misattribution. These issues result in three core limitations: noisy rewards,
low factual fidelity, and misalignment with step-level reasoning objectives. To
address these challenges, we introduce GroundedPRM, a tree-guided and
fidelity-aware framework for automatic process supervision. To reduce reward
noise and enable fine-grained credit assignment, we construct structured
reasoning paths via Monte Carlo Tree Search (MCTS). To eliminate hallucinated
supervision, we validate each intermediate step using an external tool,
providing execution-grounded correctness signals. To combine both step-level
validation and global outcome assessment, we design a hybrid reward aggregation
mechanism that fuses tool-based verification with MCTS-derived feedback.
Finally, we format the reward signal into a rationale-enhanced, generative
structure to promote interpretability and compatibility with instruction-tuned
LLMs. GroundedPRM is trained on only 40K automatically labeled samples,
amounting to just 10% of the data used by the best-performing PRM trained with
auto-labeled supervision. Nevertheless, it achieves up to a 26% relative
improvement in average performance on ProcessBench. When used for reward-guided
greedy search, GroundedPRM outperforms even PRMs trained with human-labeled
supervision, offering a scalable and verifiable path toward high-quality
process-level reasoning.
68f2675637ed56c2c926f935
Yao Zhang
68f2675637ed56c2c926f936
Yu Wu
68f2675637ed56c2c926f937
Haowei Zhang
68f2675637ed56c2c926f938
Weiguo Li
68f2675637ed56c2c926f939
Haokun Chen
68f2675637ed56c2c926f93a
Jingpei Wu
68f2675637ed56c2c926f93b
Guohao Li
68f2675637ed56c2c926f93c
Zhen Han
68f2675637ed56c2c926f93d
Volker Tresp
GroundedPRM: Tree-Guided and Fidelity-Aware Process Reward Modeling for
  Step-Level Reasoning
Process Reward Models (PRMs) aim to improve multi-step reasoning in Large
Language Models (LLMs) by supervising intermediate steps and identifying
errors. However, building effective PRMs remains challenging due to the lack of
scalable, high-quality annotations. Existing approaches rely on costly human
labeling, LLM-based self-evaluation that is prone to hallucination, or Monte
Carlo (MC) estimation, which infers step quality solely from rollout outcomes
and often introduces noisy, misaligned supervision due to credit
misattribution. These issues result in three core limitations: noisy rewards,
low factual fidelity, and misalignment with step-level reasoning objectives. To
address these challenges, we introduce GroundedPRM, a tree-guided and
fidelity-aware framework for automatic process supervision. To reduce reward
noise and enable fine-grained credit assignment, we construct structured
reasoning paths via Monte Carlo Tree Search (MCTS). To eliminate hallucinated
supervision, we validate each intermediate step using an external tool,
providing execution-grounded correctness signals. To combine both step-level
validation and global outcome assessment, we design a hybrid reward aggregation
mechanism that fuses tool-based verification with MCTS-derived feedback.
Finally, we format the reward signal into a rationale-enhanced, generative
structure to promote interpretability and compatibility with instruction-tuned
LLMs. GroundedPRM is trained on only 40K automatically labeled samples,
amounting to just 10% of the data used by the best-performing PRM trained with
auto-labeled supervision. Nevertheless, it achieves up to a 26% relative
improvement in average performance on ProcessBench. When used for reward-guided
greedy search, GroundedPRM outperforms even PRMs trained with human-labeled
supervision, offering a scalable and verifiable path toward high-quality
process-level reasoning.
GroundedPRM uses Monte Carlo Tree Search and external validation to improve multi-step reasoning in LLMs with fewer, higher-quality annotations.
Process Reward Models Large Language Models multi-step reasoning intermediate steps errors scalable annotations human labeling LLM-based self-evaluation hallucination Monte Carlo estimation credit misattribution noisy rewards low factual fidelity misalignment tree-guided fidelity-aware structured reasoning paths Monte Carlo Tree Search external tool execution-grounded correctness signals hybrid reward aggregation rationale-enhanced generative structure ProcessBench reward-guided greedy search