Free Process Rewards without Process Labels
Different from its counterpart outcome reward models (ORMs), which evaluate
the entire responses, a process reward model (PRM) scores a reasoning
trajectory step by step, providing denser and more fine grained rewards.
However, training a PRM requires labels annotated at every intermediate step,
presenting significant challenges for both manual and automatic data
collection. This paper aims to address this challenge. Both theoretically and
empirically, we show that an implicit PRM can be obtained at no
additional cost, by simply training an ORM on the cheaper response-level
labels. The only assumption is to parameterize the outcome reward as the
log-likelihood ratios of the policy and reference models, which can be
optimized regardless of the specific choice of loss objectives. In experiments,
we instantiate our implicit PRMs with various objectives and evaluate their
performance on MATH. We show that our implicit PRM outperforms a strong
MCTS-based baseline \'a la Math-Shepherd using less than 1/38 of the
training data. Its performance can be further improved with majority voting. We
further find that scaling up instructions and responses benefits our implicit
PRM, and the latter brings a larger gain. Particularly, we find that our
implicit PRM, when instantiated with the cross-entropy (CE) loss, is more
data-efficient and can keep improving generation models even when trained with
only one response per instruction, the setup that suffers from extreme data
scarcity and imbalance. Further, instructions should be relevant to downstream
tasks while the diversity of responses does not bring gains. Surprisingly,
training on extra Math-Shepherd step labels brings no further improvements to
our implicit PRM trained on only outcome data. We hope that our work will
encourage a rethinking of PRM training approaches and contribute to making
training PRMs more accessible.
674fd56a16234f46d2559385
Lifan Yuan
claimed_verified
2024-12-04T09:44:25.994Z
61506ce0b0e47650b93f79d0
/avatars/21301cd9929a1a64f3f6427413855ea8.svg
Lievan
user
lievan
674fd56a16234f46d2559386
Wendi Li
admin_assigned
2024-12-04T11:56:09.030Z
671bfaa29e5e675c7f5c4307
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/PwDA6OSSAmg6k4LliEQkZ.png
Wendi Li
user
wendili
674fd56a16234f46d2559387
Huayu Chen
674fd56a16234f46d2559388
Ganqu Cui
claimed_verified
2024-12-04T09:44:24.028Z
650eba9555dc1e841746f132
/avatars/af6f5ee78f161d25ec0afc45d2def8eb.svg
Ganqu Cui
user
ganqu
674fd56a16234f46d2559389
Ning Ding
admin_assigned
2024-12-04T11:57:00.322Z
60cf4bcb1ce3775ebb86e5d5
/avatars/12bcd18d215abf91f297f93007733148.svg
Ning Ding
user
stingning
674fd56a16234f46d255938a
Kaiyan Zhang
admin_assigned
2024-12-04T11:57:22.757Z
60bc94cd85a3ab33829b6211
/avatars/b57d36c7577fbbb42ea5b963eef4144a.svg
Kaiyan Zhang
user
iseesaw
674fd56a16234f46d255938b
Bowen Zhou
admin_assigned
2024-12-04T11:57:28.976Z
669f614b59adf5b56e05bce3
/avatars/ffd4189efbceb0e63a03db273065a44b.svg
BowenZhou
user
bowenZhou
674fd56a16234f46d255938c
Zhiyuan Liu
admin_assigned
2024-12-04T11:57:36.966Z
6472f1f7485a7c8e1bcf507b
/avatars/79dca137583875191ea883b193b630b7.svg
Zhiyuan Liu
user
zibuyu9
674fd56a16234f46d255938d
Hao Peng
Free Process Rewards without Process Labels
Different from its counterpart outcome reward models (ORMs), which evaluate
the entire responses, a process reward model (PRM) scores a reasoning
trajectory step by step, providing denser and more fine grained rewards.
However, training a PRM requires labels annotated at every intermediate step,
presenting significant challenges for both manual and automatic data
collection. This paper aims to address this challenge. Both theoretically and
empirically, we show that an implicit PRM can be obtained at no
additional cost, by simply training an ORM on the cheaper response-level
labels. The only assumption is to parameterize the outcome reward as the
log-likelihood ratios of the policy and reference models, which can be
optimized regardless of the specific choice of loss objectives. In experiments,
we instantiate our implicit PRMs with various objectives and evaluate their
performance on MATH. We show that our implicit PRM outperforms a strong
MCTS-based baseline \'a la Math-Shepherd using less than 1/38 of the
training data. Its performance can be further improved with majority voting. We
further find that scaling up instructions and responses benefits our implicit
PRM, and the latter brings a larger gain. Particularly, we find that our
implicit PRM, when instantiated with the cross-entropy (CE) loss, is more
data-efficient and can keep improving generation models even when trained with
only one response per instruction, the setup that suffers from extreme data
scarcity and imbalance. Further, instructions should be relevant to downstream
tasks while the diversity of responses does not bring gains. Surprisingly,
training on extra Math-Shepherd step labels brings no further improvements to
our implicit PRM trained on only outcome data. We hope that our work will
encourage a rethinking of PRM training approaches and contribute to making
training PRMs more accessible.
An implicit process reward model can be trained without additional step labels by parameterizing outcome rewards as log-likelihood ratios, outperforming traditional methods with less data and better generalization.
process reward model PRM outcome reward model ORM reasoning trajectory log-likelihood ratios MCTS Math-Shepherd majority voting cross-entropy loss data efficiency extreme data scarcity