A Contextual Quality Reward Model for Reliable and Efficient Best-of-N
  Sampling
Modern preference alignment techniques, such as Best-of-N (BoN) sampling,
rely on reward models trained with pairwise comparison data. While effective at
learning relative preferences, this paradigm fails to capture a signal of
response acceptability, leaving systems vulnerable to selecting the least bad
of many unacceptable options. This is particularly problematic for hard
prompts, where the risk of such false acceptances increases with the number of
samples. In this paper, we address this critical reliability gap by introducing
a new data collection and modeling framework. By augmenting preference data
with an outside option, inspired by discrete choice models, we train a reward
model that can distinguish not just what is better, but what is
good enough. We leverage this capability to create an adaptive
inference strategy, best of mini-N in-loop, which partitions the generation
budget into sequential loops with a calibrated, early-exit condition. Our
experiments show that when tuned as an alignment guardrail, it reduces
reliability failures by 70\%, and when tuned as an inference accelerator, it
improves average inference speed by over 22\% in IMDB-sentiment setting. We
thus provide a principled and flexible framework for practitioners to
explicitly manage the trade-off between reliability and computational
efficiency.
68e4715be4e093a7044e4cce
Hyung Gyu Rho
claimed_verified
2025-10-07T12:27:19.479Z
6642dafed48363a46ddb69ed
/avatars/111fb26ea38fb2e6f8470f7ed513d48d.svg
hyung gyu rho
user
sirano1004
A Contextual Quality Reward Model for Reliable and Efficient Best-of-N
  Sampling
Modern preference alignment techniques, such as Best-of-N (BoN) sampling,
rely on reward models trained with pairwise comparison data. While effective at
learning relative preferences, this paradigm fails to capture a signal of
response acceptability, leaving systems vulnerable to selecting the least bad
of many unacceptable options. This is particularly problematic for hard
prompts, where the risk of such false acceptances increases with the number of
samples. In this paper, we address this critical reliability gap by introducing
a new data collection and modeling framework. By augmenting preference data
with an outside option, inspired by discrete choice models, we train a reward
model that can distinguish not just what is better, but what is
good enough. We leverage this capability to create an adaptive
inference strategy, best of mini-N in-loop, which partitions the generation
budget into sequential loops with a calibrated, early-exit condition. Our
experiments show that when tuned as an alignment guardrail, it reduces
reliability failures by 70\%, and when tuned as an inference accelerator, it
improves average inference speed by over 22\% in IMDB-sentiment setting. We
thus provide a principled and flexible framework for practitioners to
explicitly manage the trade-off between reliability and computational
efficiency.
A new framework using an outside option in preference data collection and modeling improves reliability and efficiency in preference alignment techniques.
Best-of-N (BoN) sampling reward models pairwise comparison data discrete choice models adaptive inference strategy best of mini-N in-loop reliability failures inference speed IMDB-sentiment setting