All Proxy Rewards are Bad, Can We Hedge to Make Some Useful?
Hadi Khalaf, Claudio Mayrink Verdun, Alex Oesterling, Himabindu Lakkaraju, Flavio Calmon
A common paradigm to improve the performance of large language models is optimizing for a reward model. Reward models assign a numerical score to LLM outputs indicating, for example, which response would likely be preferred by a user or is most aligned with safety goals.  However, reward models are never perfect. They inevitably function as proxies for  complex desiderata such as correctness, helpfulness, and safety. By overoptimizing for a misspecified reward, we can subvert intended alignment goals and reduce overall performance - a phenomenon commonly referred to as reward hacking. In this work, we characterize reward hacking in inference-time alignment and demonstrate when and how we can mitigate it by hedging on the proxy reward. Hedging represents a tactical choice to avoid placing undue confidence in high but potentially misleading proxy reward signals. We study reward hacking under Best-of-$n$ (BoN) sampling, along with two inference-time methods, namely a novel method Best-of-Poisson (BoP) and Soft-Best-of-$n$ (SBoN), which introduce a parameter to control our confidence in the reward. We then propose $\texttt{HedgeTune}$ as an efficient algorithm to find the optimal hedging parameter. We demonstrate through experiments that hedging mitigates reward hacking and achieves superior distortion-reward tradeoffs with minimal computational overhead.