Rethinking Reward Models for Multi-Domain Test-Time Scaling
The reliability of large language models (LLMs) during test-time scaling is
often assessed with external verifiers or reward models that
distinguish correct reasoning from flawed logic. Prior work generally assumes
that process reward models (PRMs), which score every intermediate reasoning
step, outperform outcome reward models (ORMs) that assess only the final
answer. This view is based mainly on evidence from narrow, math-adjacent
domains. We present the first unified evaluation of four reward model variants,
discriminative ORM and PRM (\DisORM, \DisPRM) and generative ORM and PRM
(\GenORM, \GenPRM), across 14 diverse domains. Contrary to conventional wisdom,
we find that (i) \DisORM performs on par with \DisPRM, (ii) \GenPRM is not
competitive, and (iii) overall, \GenORM is the most robust, yielding
significant and consistent gains across every tested domain. We attribute this
to PRM-style stepwise scoring, which inherits label noise from LLM
auto-labeling and has difficulty evaluating long reasoning trajectories,
including those involving self-correcting reasoning. Our theoretical analysis
shows that step-wise aggregation compounds errors as reasoning length grows,
and our empirical observations confirm this effect. These findings challenge
the prevailing assumption that fine-grained supervision is always better and
support generative outcome verification for multi-domain deployment. We
publicly release our code, datasets, and checkpoints at
https://github.com/db-Lee/Multi-RM{\small\texttt{https://github.com/db-Lee/Multi-RM}}
to facilitate future research in multi-domain settings.
68ddd3816024653e8a3ed052
Dong Bok Lee
claimed_verified
2025-10-07T12:29:12.606Z
64f000769e7770db74d44bba
/avatars/d015820380ffb823b1b35df64dcd3457.svg
Dong-Bok Lee
user
dongboklee
68ddd3816024653e8a3ed053
Seanie Lee
68ddd3816024653e8a3ed054
Sangwoo Park
claimed_verified
2025-10-02T13:55:49.672Z
638716c14e00d7fc0902fef4
/avatars/5fa8152f8c0e4e600d1a64802c3e0103.svg
Sangwoo Park
user
Sangsang
68ddd3816024653e8a3ed055
Minki Kang
68ddd3816024653e8a3ed056
Jinheon Baek
68ddd3816024653e8a3ed057
Dongki Kim
68ddd3816024653e8a3ed058
Dominik Wagner
claimed_verified
2025-10-02T13:55:46.164Z
6311ba6f05cc08a1408d910a
https://cdn-avatars.huggingface.co/v1/production/uploads/1662997515866-6311ba6f05cc08a1408d910a.png
Dominik Wagner
user
dwgnr
68ddd3816024653e8a3ed059
Jiongdao Jin
claimed_verified
2025-10-02T13:55:53.571Z
65f06c8356cb8a32b41baf83
/avatars/ac66cc63f3abade4a859f7bf9357682a.svg
Jiongdao Jin
user
jiongdao
68ddd3816024653e8a3ed05a
Heejun Lee
68ddd3816024653e8a3ed05b
Tobias Bocklet
68ddd3816024653e8a3ed05c
Jinyu Wang
68ddd3816024653e8a3ed05d
Jingjing Fu
68ddd3816024653e8a3ed05e
Sung Ju Hwang
68ddd3816024653e8a3ed05f
Jiang Bia
68ddd3816024653e8a3ed060
Lei Song
Rethinking Reward Models for Multi-Domain Test-Time Scaling
The reliability of large language models (LLMs) during test-time scaling is
often assessed with external verifiers or reward models that
distinguish correct reasoning from flawed logic. Prior work generally assumes
that process reward models (PRMs), which score every intermediate reasoning
step, outperform outcome reward models (ORMs) that assess only the final
answer. This view is based mainly on evidence from narrow, math-adjacent
domains. We present the first unified evaluation of four reward model variants,
discriminative ORM and PRM (\DisORM, \DisPRM) and generative ORM and PRM
(\GenORM, \GenPRM), across 14 diverse domains. Contrary to conventional wisdom,
we find that (i) \DisORM performs on par with \DisPRM, (ii) \GenPRM is not
competitive, and (iii) overall, \GenORM is the most robust, yielding
significant and consistent gains across every tested domain. We attribute this
to PRM-style stepwise scoring, which inherits label noise from LLM
auto-labeling and has difficulty evaluating long reasoning trajectories,
including those involving self-correcting reasoning. Our theoretical analysis
shows that step-wise aggregation compounds errors as reasoning length grows,
and our empirical observations confirm this effect. These findings challenge
the prevailing assumption that fine-grained supervision is always better and
support generative outcome verification for multi-domain deployment. We
publicly release our code, datasets, and checkpoints at
https://github.com/db-Lee/Multi-RM{\small\texttt{https://github.com/db-Lee/Multi-RM}}
to facilitate future research in multi-domain settings.
A unified evaluation across diverse domains shows that generative outcome reward models outperform process reward models and discriminative outcome reward models in assessing large language model reliability.
large language models external verifiers reward models process reward models outcome reward models discriminative ORM discriminative PRM generative ORM generative PRM label noise self-correcting reasoning step-wise aggregation multi-domain deployment