DuaShepherd: Integrating Stepwise Correctness and Potential Rewards for
  Mathematical Reasoning
In this paper, we propose DuaShepherd, a novel reward modeling framework that
integrates two complementary reward signals, correctness and potential, to
enhance the mathematical reasoning capabilities of Large Language Models
(LLMs). While correctness-based signals emphasize identification of stepwise
errors, potential-based signals focus on the likelihood of reaching the correct
final answer. We developed an automated pipeline for constructing large-scale
reward modeling dataset with both signals. A unified, multi-head architecture
was explored to train the two reward models in a multi-task setup,
demonstrating benefits from learning both correctness and potential in
parallel. By combining these two signals into a compound probability, our model
achieves consistent performance improvements across multiple benchmarks.
Empirical evaluations on MATH500 and ProcessBench confirm that this combined
reward significantly outperforms models trained on either reward type alone,
achieving state-of-the-art performance under comparable resource constraints.
685ad3e8c9f2cb1e24e9c4b3
Yuanhao Wu
claimed_verified
2025-06-27T20:31:12.729Z
61fbe895869c51c62902e9c4
/avatars/c7dc13c025a2b0e33a762bde01eac5f3.svg
Yuanhao Wu
user
wuyhthu
685ad3e8c9f2cb1e24e9c4b4
Juntong Song
685ad3e8c9f2cb1e24e9c4b5
Hanning Zhang
685ad3e8c9f2cb1e24e9c4b6
Tong Zhang
685ad3e8c9f2cb1e24e9c4b7
Cheng Niu
DuaShepherd: Integrating Stepwise Correctness and Potential Rewards for
  Mathematical Reasoning
In this paper, we propose DuaShepherd, a novel reward modeling framework that
integrates two complementary reward signals, correctness and potential, to
enhance the mathematical reasoning capabilities of Large Language Models
(LLMs). While correctness-based signals emphasize identification of stepwise
errors, potential-based signals focus on the likelihood of reaching the correct
final answer. We developed an automated pipeline for constructing large-scale
reward modeling dataset with both signals. A unified, multi-head architecture
was explored to train the two reward models in a multi-task setup,
demonstrating benefits from learning both correctness and potential in
parallel. By combining these two signals into a compound probability, our model
achieves consistent performance improvements across multiple benchmarks.
Empirical evaluations on MATH500 and ProcessBench confirm that this combined
reward significantly outperforms models trained on either reward type alone,
achieving state-of-the-art performance under comparable resource constraints.
A novel reward modeling framework DuaShepherd integrates correctness and potential signals into a unified multi-head architecture to enhance LLMs' mathematical reasoning capabilities and achieve state-of-the-art performance.
reward modeling Large Language Models (LLMs) correctness signals potential signals automated pipeline multi-head architecture multi-task setup compound probability MATH500 ProcessBench