Reward Models Enable Scalable Code Verification by Trading Accuracy for
  Throughput
The standard paradigm for solving coding tasks via large language models
(LLMs) is to generate-then-rank programs, where the latter step uses a verifier
in the ranking process. The growing consensus is that a comprehensive verifier
(e.g., a full test suite) should be prioritized over an outcome reward model
(ORM) whenever possible, with little consideration given to the trade-offs
involved. We aim to challenge this assumption by systematically exploring the
tradeoff between speed and accuracy. We find that ORMs play a crucial role in
scaling verification through trading accuracy for speed, even when a
comprehensive verifier is available. Their value becomes especially apparent
when used in a generate-prune-then-rank approach, where a faster but less
accurate verifier removes incorrect solutions prior to ranking -- leading to a
system that is 11.65x faster while only being 8.33% less accurate than the full
test suite. We analyze the generate-prune-then-rank approach and show that it
works by filtering out incorrect but highly ranked solutions. These findings
enable the design of scalable and accurate program ranking systems.
6850651a2932e11c891b5892
Gabriel Orlanski
claimed_verified
2025-06-17T21:13:24.275Z
603ac96345ca069a4c0817e3
/avatars/54c62f1ab3bfeded8a1e73b0a1b38768.svg
Gabe Orlanski
user
gabeorlanski
6850651a2932e11c891b5893
Nicholas Roberts
6850651a2932e11c891b5894
Aws Albarghouthi
6850651a2932e11c891b5895
Frederic Sala
Reward Models Enable Scalable Code Verification by Trading Accuracy for
  Throughput
The standard paradigm for solving coding tasks via large language models
(LLMs) is to generate-then-rank programs, where the latter step uses a verifier
in the ranking process. The growing consensus is that a comprehensive verifier
(e.g., a full test suite) should be prioritized over an outcome reward model
(ORM) whenever possible, with little consideration given to the trade-offs
involved. We aim to challenge this assumption by systematically exploring the
tradeoff between speed and accuracy. We find that ORMs play a crucial role in
scaling verification through trading accuracy for speed, even when a
comprehensive verifier is available. Their value becomes especially apparent
when used in a generate-prune-then-rank approach, where a faster but less
accurate verifier removes incorrect solutions prior to ranking -- leading to a
system that is 11.65x faster while only being 8.33% less accurate than the full
test suite. We analyze the generate-prune-then-rank approach and show that it
works by filtering out incorrect but highly ranked solutions. These findings
enable the design of scalable and accurate program ranking systems.