Controlled Decoding from Language Models
We propose controlled decoding (CD), a novel off-policy reinforcement
learning method to control the autoregressive generation from language models
towards high reward outcomes. CD solves an off-policy reinforcement learning
problem through a value function for the reward, which we call a prefix scorer.
The prefix scorer is used at inference time to steer the generation towards
higher reward outcomes. We show that the prefix scorer may be trained on
(possibly) off-policy data to predict the expected reward when decoding is
continued from a partially decoded response. We empirically demonstrate that CD
is effective as a control mechanism on Reddit conversations corpus. We also
show that the modularity of the design of CD makes it possible to control for
multiple rewards, effectively solving a multi-objective reinforcement learning
problem with no additional complexity. Finally, we show that CD can be applied
in a novel blockwise fashion at inference-time, again without the need for any
training-time changes, essentially bridging the gap between the popular
best-of-K strategy and token-level reinforcement learning. This makes CD a
promising approach for alignment of language models.
653b17660157cf1d9c1261bb
Sidharth Mudgal
653b17660157cf1d9c1261bc
Jong Lee
653b17660157cf1d9c1261bd
Harish Ganapathy
653b17660157cf1d9c1261be
YaGuang Li
claimed_verified
2023-10-27T18:43:14.777Z
642f957c4e7b53c5addcae64
/avatars/0bb13eafdc0a80e1d030d570efaa7033.svg
YaGuang Li
user
kakaro0t
653b17660157cf1d9c1261bf
Tao Wang
653b17660157cf1d9c1261c0
Yanping Huang
653b17660157cf1d9c1261c1
Zhifeng Chen
653b17660157cf1d9c1261c2
Heng-Tze Cheng
653b17660157cf1d9c1261c3
Michael Collins
653b17660157cf1d9c1261c4
Trevor Strohman
653b17660157cf1d9c1261c5
Jilin Chen
653b17660157cf1d9c1261c6
Alex Beutel
admin_assigned
2023-10-27T09:27:56.647Z
636dc726ffbe479c979c3046
/avatars/287f422bb0c5a26d8d6c0649ac83ba93.svg
Alex Beutel
user
alexbeutel
653b17660157cf1d9c1261c7
Ahmad Beirami
claimed_verified
2023-11-06T10:57:30.766Z
65482d45407bb19ff5adb8a4
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/_etKjaaubh3osxp3R2MP7.jpeg
Ahmad Beirami
user
beirami
Controlled Decoding from Language Models
We propose controlled decoding (CD), a novel off-policy reinforcement
learning method to control the autoregressive generation from language models
towards high reward outcomes. CD solves an off-policy reinforcement learning
problem through a value function for the reward, which we call a prefix scorer.
The prefix scorer is used at inference time to steer the generation towards
higher reward outcomes. We show that the prefix scorer may be trained on
(possibly) off-policy data to predict the expected reward when decoding is
continued from a partially decoded response. We empirically demonstrate that CD
is effective as a control mechanism on Reddit conversations corpus. We also
show that the modularity of the design of CD makes it possible to control for
multiple rewards, effectively solving a multi-objective reinforcement learning
problem with no additional complexity. Finally, we show that CD can be applied
in a novel blockwise fashion at inference-time, again without the need for any
training-time changes, essentially bridging the gap between the popular
best-of-K strategy and token-level reinforcement learning. This makes CD a
promising approach for alignment of language models.
Controlled decoding is an off-policy reinforcement learning method that uses a prefix scorer to guide language model generation towards high rewards and can handle multiple objectives without additional complexity.
controlled decoding off-policy reinforcement learning prefix scorer autoregressive generation Reddit conversations corpus multi-objective reinforcement learning best-of-$K$ token-level reinforcement learning