Fast Best-of-N Decoding via Speculative Rejection
The safe and effective deployment of Large Language Models (LLMs) involves a
critical step called alignment, which ensures that the model's responses are in
accordance with human preferences. Prevalent alignment techniques, such as DPO,
PPO and their variants, align LLMs by changing the pre-trained model weights
during a phase called post-training. While predominant, these post-training
methods add substantial complexity before LLMs can be deployed. Inference-time
alignment methods avoid the complex post-training step and instead bias the
generation towards responses that are aligned with human preferences. The
best-known inference-time alignment method, called Best-of-N, is as effective
as the state-of-the-art post-training procedures. Unfortunately, Best-of-N
requires vastly more resources at inference time than standard decoding
strategies, which makes it computationally not viable. In this work, we
introduce Speculative Rejection, a computationally-viable inference-time
alignment algorithm. It generates high-scoring responses according to a given
reward model, like Best-of-N does, while being between 16 to 32 times more
computationally efficient.
6720872c71d10b163b13e299
Hanshi Sun
claimed_verified
2024-10-30T09:53:08.123Z
651b23a0a6e00a1678c9d8af
https://cdn-avatars.huggingface.co/v1/production/uploads/651b23a0a6e00a1678c9d8af/_Ip-ysvcjGpFCI9HtcTgx.jpeg
Hanshi
user
preminstrel
6720872c71d10b163b13e29a
Momin Haider
6720872c71d10b163b13e29b
Ruiqi Zhang
6720872c71d10b163b13e29c
Huitao Yang
6720872c71d10b163b13e29d
Jiahao Qiu
claimed_verified
2025-05-28T19:15:24.394Z
65271df5abd7795aaa1fd86e
/avatars/99e1aabaefcf5e7f438817375cd1ddef.svg
Jiahao Qiu
user
jiahaoq
6720872c71d10b163b13e29e
Ming Yin
6720872c71d10b163b13e29f
Mengdi Wang
6720872c71d10b163b13e2a0
Peter Bartlett
6720872c71d10b163b13e2a1
Andrea Zanette
Fast Best-of-N Decoding via Speculative Rejection
The safe and effective deployment of Large Language Models (LLMs) involves a
critical step called alignment, which ensures that the model's responses are in
accordance with human preferences. Prevalent alignment techniques, such as DPO,
PPO and their variants, align LLMs by changing the pre-trained model weights
during a phase called post-training. While predominant, these post-training
methods add substantial complexity before LLMs can be deployed. Inference-time
alignment methods avoid the complex post-training step and instead bias the
generation towards responses that are aligned with human preferences. The
best-known inference-time alignment method, called Best-of-N, is as effective
as the state-of-the-art post-training procedures. Unfortunately, Best-of-N
requires vastly more resources at inference time than standard decoding
strategies, which makes it computationally not viable. In this work, we
introduce Speculative Rejection, a computationally-viable inference-time
alignment algorithm. It generates high-scoring responses according to a given
reward model, like Best-of-N does, while being between 16 to 32 times more
computationally efficient.
Speculative Rejection is a computationally efficient inference-time alignment method for Large Language Models, offering improved performance over Best-of-N with reduced resource requirements.
Large Language Models LLMs alignment DPO PPO post-training inference-time alignment Best-of-N reward model Speculative Rejection