Sample-Efficient Alignment for LLMs
We study methods for efficiently aligning large language models (LLMs) with
human preferences given budgeted online feedback. We first formulate the LLM
alignment problem in the frame of contextual dueling bandits. This formulation,
subsuming recent paradigms such as online RLHF and online DPO, inherently
quests for sample-efficient algorithms that incorporate online active
exploration. Leveraging insights from bandit theory, we introduce a unified
algorithm based on Thompson sampling and highlight its applications in two
distinct LLM alignment scenarios. The practical agent that efficiently
implements this algorithm, named SEA (Sample-Efficient Alignment), is
empirically validated through extensive experiments across three model scales
(1B, 2.8B, 6.9B) and three preference learning algorithms (DPO, IPO, SLiC). The
results demonstrate that SEA achieves highly sample-efficient alignment with
oracle's preferences, outperforming recent active exploration methods for LLMs.
Additionally, we release the implementation of SEA together with an efficient
codebase designed for online alignment of LLMs, aiming to accelerate future
research in this field.
672ad09444b2453edfd339a7
Zichen Liu
claimed_verified
2024-11-15T09:37:21.730Z
65f5392c68b8e0cb3c9977a2
/avatars/aa64772475098e8a135c13072fde6744.svg
Zichen
user
lkevinzc
672ad09444b2453edfd339a8
Changyu Chen
claimed_verified
2024-11-15T09:37:19.497Z
64e416dc54e18f390ef79ba4
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/5n01J00ZaVRrebsON8iYA.jpeg
Changyu Chen
user
Cameron-Chen
672ad09444b2453edfd339a9
Chao Du
claimed_verified
2024-11-15T09:37:17.255Z
632407c892e07e3ca20aca28
/avatars/23b51b37b12b51a0947f687d1de4d3b5.svg
Chao Du
user
duchao
672ad09444b2453edfd339aa
Wee Sun Lee
672ad09444b2453edfd339ab
Min Lin
Sample-Efficient Alignment for LLMs
We study methods for efficiently aligning large language models (LLMs) with
human preferences given budgeted online feedback. We first formulate the LLM
alignment problem in the frame of contextual dueling bandits. This formulation,
subsuming recent paradigms such as online RLHF and online DPO, inherently
quests for sample-efficient algorithms that incorporate online active
exploration. Leveraging insights from bandit theory, we introduce a unified
algorithm based on Thompson sampling and highlight its applications in two
distinct LLM alignment scenarios. The practical agent that efficiently
implements this algorithm, named SEA (Sample-Efficient Alignment), is
empirically validated through extensive experiments across three model scales
(1B, 2.8B, 6.9B) and three preference learning algorithms (DPO, IPO, SLiC). The
results demonstrate that SEA achieves highly sample-efficient alignment with
oracle's preferences, outperforming recent active exploration methods for LLMs.
Additionally, we release the implementation of SEA together with an efficient
codebase designed for online alignment of LLMs, aiming to accelerate future
research in this field.
A sample-efficient algorithm called SEA is introduced for aligning large language models with human preferences using contextual dueling bandits and Thompson sampling.
contextual dueling bandits online RLHF online DPO Thompson sampling sample-efficient active exploration DPO IPO SLiC online alignment