Reward-Augmented Decoding: Efficient Controlled Text Generation With a
  Unidirectional Reward Model
While large language models have proven effective in a huge range of
downstream applications, they often generate text that is problematic or lacks
a desired attribute. In this paper, we introduce Reward-Augmented Decoding
(RAD), a text generation procedure that uses a small unidirectional reward
model to encourage a language model to generate text that has certain
properties. Specifically, RAD uses the reward model to score generations as
they are produced and rescales sampling probabilities to favor high-reward
tokens. By using a unidirectional reward model, RAD can cache activations from
prior generation steps to decrease computational overhead. Through experiments
on generating non-toxic and sentiment-controlled text, we demonstrate that RAD
performs best among methods that change only the generation procedure and
matches the performance of state-of-the-art methods that involve re-training
the language model. We further validate that RAD is effective on very large
language models while incurring a minimal computational overhead.
652deaa839b7a1b6c84fe9a0
Haikang Deng
extracted_confirmed
2023-10-17T05:35:45.468Z
624cf6ba5ee9cb3d45588971
/avatars/f0fc5e353951d3e501542dab559bbdb5.svg
Haikang Deng
user
hk
652deaa839b7a1b6c84fe9a1
Colin Raffel
extracted_pending
2023-10-17T02:00:09.545Z
6079c29765b9d0165cb18392
https://cdn-avatars.huggingface.co/v1/production/uploads/1618592397610-noauth.jpeg
Colin Raffel
user
craffel
Reward-Augmented Decoding: Efficient Controlled Text Generation With a
  Unidirectional Reward Model
While large language models have proven effective in a huge range of
downstream applications, they often generate text that is problematic or lacks
a desired attribute. In this paper, we introduce Reward-Augmented Decoding
(RAD), a text generation procedure that uses a small unidirectional reward
model to encourage a language model to generate text that has certain
properties. Specifically, RAD uses the reward model to score generations as
they are produced and rescales sampling probabilities to favor high-reward
tokens. By using a unidirectional reward model, RAD can cache activations from
prior generation steps to decrease computational overhead. Through experiments
on generating non-toxic and sentiment-controlled text, we demonstrate that RAD
performs best among methods that change only the generation procedure and
matches the performance of state-of-the-art methods that involve re-training
the language model. We further validate that RAD is effective on very large
language models while incurring a minimal computational overhead.
Reward-Augmented Decoding (RAD) enhances text generation by using a unidirectional reward model to favor desirable text attributes, achieving performance comparable to re-training methods with minimal computational cost.
Reward-Augmented Decoding RAD reward model unidirectional caching computational overhead non-toxic text sentiment-controlled text large language models