Offline Regularised Reinforcement Learning for Large Language Models
  Alignment
The dominant framework for alignment of large language models (LLM), whether
through reinforcement learning from human feedback or direct preference
optimisation, is to learn from preference data. This involves building datasets
where each element is a quadruplet composed of a prompt, two independent
responses (completions of the prompt) and a human preference between the two
independent responses, yielding a preferred and a dis-preferred response. Such
data is typically scarce and expensive to collect. On the other hand,
single-trajectory datasets where each element is a triplet composed of a
prompt, a response and a human feedback is naturally more abundant. The
canonical element of such datasets is for instance an LLM's response to a
user's prompt followed by a user's feedback such as a thumbs-up/down.
Consequently, in this work, we propose DRO, or Direct Reward
Optimisation, as a framework and associated algorithms that do not require
pairwise preferences. DRO uses a simple mean-squared objective that can be
implemented in various ways. We validate our findings empirically, using T5
encoder-decoder language models, and show DRO's performance over selected
baselines such as Kahneman-Tversky Optimization (KTO). Thus, we confirm that
DRO is a simple and empirically compelling method for single-trajectory policy
optimisation.
665888d824f3d17e4bdc9c89
Pierre Harvey Richemond
admin_assigned
2024-05-31T11:07:07.227Z
636eeea187545ca5a13bedbc
/avatars/cdbb5e0a700b60a43c99f93f8e9ef2b4.svg
Pierre Richemond
user
PierreRichemond
665888d824f3d17e4bdc9c8a
Yunhao Tang
665888d824f3d17e4bdc9c8b
Daniel Guo
665888d824f3d17e4bdc9c8c
Daniele Calandriello
665888d824f3d17e4bdc9c8d
Mohammad Gheshlaghi Azar
665888d824f3d17e4bdc9c8e
Rafael Rafailov
admin_assigned
2024-05-31T11:07:57.576Z
64701c98d742e9ef6522598f
/avatars/df3a52bd4bc374d7cb4061f449f75c07.svg
Rafael Rafailov
user
rmrafailov
665888d824f3d17e4bdc9c8f
Bernardo Avila Pires
665888d824f3d17e4bdc9c90
Eugene Tarassov
claimed_verified
2024-05-31T14:24:11.912Z
63280235bfc72a1d59d029c3
/avatars/85be03ddbe48a0d7a12f8fd95eeb00a1.svg
Eugene Tarassov
user
etarassov
665888d824f3d17e4bdc9c91
Lucas Spangher
admin_assigned
2024-05-31T11:08:28.512Z
64d6785736b05ef8ffcf8d63
/avatars/7bb694f2f5bf55462fe8d954b48a6a0e.svg
Lucas Spangher
user
Lucas-mit-psfc
665888d824f3d17e4bdc9c92
Will Ellsworth
665888d824f3d17e4bdc9c93
Aliaksei Severyn
665888d824f3d17e4bdc9c94
Jonathan Mallinson
665888d824f3d17e4bdc9c95
Lior Shani
665888d824f3d17e4bdc9c96
Gil Shamir
665888d824f3d17e4bdc9c97
Rishabh Joshi
665888d824f3d17e4bdc9c98
Tianqi Liu
admin_assigned
2024-05-31T11:09:13.411Z
646784af696e7355f5d5001f
/avatars/39ce8999c296584ee0bdeb7848eee6d9.svg
Tianqi Liu
user
TianqiLiuAI
665888d824f3d17e4bdc9c99
Remi Munos
665888d824f3d17e4bdc9c9a
Bilal Piot
extracted_pending
2025-02-03T18:28:18.402Z
67a10a285685d37e288d1ed0
/avatars/456dcb7bfe61b2bb423b89c9d622c54c.svg
Bilal Piot
user
bilalpiot
Offline Regularised Reinforcement Learning for Large Language Models
  Alignment
The dominant framework for alignment of large language models (LLM), whether
through reinforcement learning from human feedback or direct preference
optimisation, is to learn from preference data. This involves building datasets
where each element is a quadruplet composed of a prompt, two independent
responses (completions of the prompt) and a human preference between the two
independent responses, yielding a preferred and a dis-preferred response. Such
data is typically scarce and expensive to collect. On the other hand,
single-trajectory datasets where each element is a triplet composed of a
prompt, a response and a human feedback is naturally more abundant. The
canonical element of such datasets is for instance an LLM's response to a
user's prompt followed by a user's feedback such as a thumbs-up/down.
Consequently, in this work, we propose DRO, or Direct Reward
Optimisation, as a framework and associated algorithms that do not require
pairwise preferences. DRO uses a simple mean-squared objective that can be
implemented in various ways. We validate our findings empirically, using T5
encoder-decoder language models, and show DRO's performance over selected
baselines such as Kahneman-Tversky Optimization (KTO). Thus, we confirm that
DRO is a simple and empirically compelling method for single-trajectory policy
optimisation.
Direct Reward Optimisation (DRO) framework uses single-trajectory datasets for large language model alignment, offering a simpler and effective alternative to preference-based methods.
reinforcement learning from human feedback direct preference optimisation preference data quadruplet triplet human preference Abundant Direct Reward Optimisation mean-squared objective T5 encoder-decoder Kahneman-Tversky Optimization single-trajectory policy optimisation