AdaptiveStep: Automatically Dividing Reasoning Step through Model
  Confidence
Current approaches for training Process Reward Models (PRMs) often involve
breaking down responses into multiple reasoning steps using rule-based
techniques, such as using predefined placeholder tokens or setting the
reasoning step's length into a fixed size. These approaches overlook the fact
that specific words do not typically mark true decision points in a text. To
address this, we propose AdaptiveStep, a method that divides reasoning steps
based on the model's confidence in predicting the next word. This division
method provides more decision-making information at each step, enhancing
downstream tasks, such as reward model learning. Moreover, our method does not
require manual annotation. We demonstrate its effectiveness through experiments
with AdaptiveStep-trained PRMs in mathematical reasoning and code generation
tasks. Experimental results indicate that the outcome PRM achieves
state-of-the-art Best-of-N performance, surpassing greedy search strategy with
token-level value-guided decoding, while also reducing construction costs by
over 30% compared to existing open-source PRMs. In addition, we provide a
thorough analysis and case study on the PRM's performance, transferability, and
generalization capabilities.
67b6a9a7c721bee91cac2888
Yuliang Liu
admin_assigned
2025-02-20T17:11:40.282Z
6529f79e802e3d1a4f8ec662
/avatars/d05320c370a6497d8792ef5acb563dd5.svg
Yuliang Liu
user
yuliang03181
67b6a9a7c721bee91cac2889
Junjie Lu
claimed_verified
2025-02-24T13:05:14.854Z
660e1bb6e089e44d2906a7e8
https://cdn-avatars.huggingface.co/v1/production/uploads/660e1bb6e089e44d2906a7e8/MJU8nHQ0xgiPzO8J6L8EC.jpeg
Junjie Lu
user
Lux0926
67b6a9a7c721bee91cac288a
Zhaoling Chen
67b6a9a7c721bee91cac288b
Chaofeng Qu
admin_assigned
2025-02-20T17:11:10.302Z
65d0ae74c4d2b2e40281256e
https://cdn-avatars.huggingface.co/v1/production/uploads/65d0ae74c4d2b2e40281256e/nNTJkAkHsu9Fru4CpRphC.jpeg
Chaofeng QU
user
kylebrovloski
67b6a9a7c721bee91cac288c
Jason Klein Liu
admin_assigned
2025-02-20T17:11:16.565Z
6731c04d5f55903a1d8c307c
/avatars/704b1d628c55e3141194b08736f21267.svg
Jason Klein Liu
user
jasonkleinlove
67b6a9a7c721bee91cac288d
Chonghan Liu
67b6a9a7c721bee91cac288e
Zefan Cai
67b6a9a7c721bee91cac288f
Yunhui Xia
67b6a9a7c721bee91cac2890
Li Zhao
67b6a9a7c721bee91cac2891
Jiang Bian
admin_assigned
2025-02-20T17:00:08.549Z
63f253f8f4e30ffd2bd308fb
/avatars/303f4c7ee588f638acf78a7966786e1e.svg
Jiang Bian
user
bianjiang
67b6a9a7c721bee91cac2892
Chuheng Zhang
admin_assigned
2025-02-20T16:59:49.312Z
64c32727c370f29a10334d35
/avatars/a3284bdd2e51f437433b89a96a904448.svg
ZhangChuheng
user
zhangchuheng123
67b6a9a7c721bee91cac2893
Wei Shen
67b6a9a7c721bee91cac2894
Zhouhan Lin
AdaptiveStep: Automatically Dividing Reasoning Step through Model
  Confidence
Current approaches for training Process Reward Models (PRMs) often involve
breaking down responses into multiple reasoning steps using rule-based
techniques, such as using predefined placeholder tokens or setting the
reasoning step's length into a fixed size. These approaches overlook the fact
that specific words do not typically mark true decision points in a text. To
address this, we propose AdaptiveStep, a method that divides reasoning steps
based on the model's confidence in predicting the next word. This division
method provides more decision-making information at each step, enhancing
downstream tasks, such as reward model learning. Moreover, our method does not
require manual annotation. We demonstrate its effectiveness through experiments
with AdaptiveStep-trained PRMs in mathematical reasoning and code generation
tasks. Experimental results indicate that the outcome PRM achieves
state-of-the-art Best-of-N performance, surpassing greedy search strategy with
token-level value-guided decoding, while also reducing construction costs by
over 30% compared to existing open-source PRMs. In addition, we provide a
thorough analysis and case study on the PRM's performance, transferability, and
generalization capabilities.
AdaptiveStep Process Reward Models (PRMs) reasoning steps rule-based techniques placeholder tokens confidence in predicting the next word downstream tasks reward model learning Best-of-N performance greedy search strategy token-level value-guided decoding construction costs performance analysis transferability generalization capabilities