LiPO: Listwise Preference Optimization through Learning-to-Rank
Aligning language models (LMs) with curated human feedback is critical to
control their behaviors in real-world applications. Several recent policy
optimization methods, such as DPO and SLiC, serve as promising alternatives to
the traditional Reinforcement Learning from Human Feedback (RLHF) approach. In
practice, human feedback often comes in a format of a ranked list over multiple
responses to amortize the cost of reading prompt. Multiple responses can also
be ranked by reward models or AI feedback. There lacks such a study on directly
fitting upon a list of responses. In this work, we formulate the LM alignment
as a listwise ranking problem and describe the Listwise Preference Optimization
(LiPO) framework, where the policy can potentially learn more effectively from
a ranked list of plausible responses given the prompt. This view draws an
explicit connection to Learning-to-Rank (LTR), where most existing preference
optimization work can be mapped to existing ranking objectives, especially
pairwise ones. Following this connection, we provide an examination of ranking
objectives that are not well studied for LM alignment withDPO and SLiC as
special cases when list size is two. In particular, we highlight a specific
method, LiPO-{\lambda}, which leverages a state-of-the-art listwise ranking
objective and weights each preference pair in a more advanced manner. We show
that LiPO-{\lambda} can outperform DPO and SLiC by a clear margin on two
preference alignment tasks.
65c1b67a0ad80fd1f69b12a7
Tianqi Liu
extracted_confirmed
2024-02-28T19:29:33.612Z
646784af696e7355f5d5001f
/avatars/39ce8999c296584ee0bdeb7848eee6d9.svg
Tianqi Liu
user
TianqiLiuAI
65c1b67a0ad80fd1f69b12a8
Zhen Qin
extracted_confirmed
2024-06-18T08:17:53.197Z
64a2bd9245dde1d6ac7b4d81
/avatars/8a238a832bfcf14e7704a2152b254a0e.svg
Zhen Qin
user
paulqin
65c1b67a0ad80fd1f69b12a9
Junru Wu
65c1b67a0ad80fd1f69b12aa
Jiaming Shen
65c1b67a0ad80fd1f69b12ab
Misha Khalman
claimed_verified
2024-02-07T20:41:05.847Z
65c2ad2790703d2dbcf79471
https://cdn-avatars.huggingface.co/v1/production/uploads/65c2ad2790703d2dbcf79471/H15_ASSI7fCFDCDF4xgIb.jpeg
Misha Khalman
user
khalman
65c1b67a0ad80fd1f69b12ac
Rishabh Joshi
65c1b67a0ad80fd1f69b12ad
Yao Zhao
65c1b67a0ad80fd1f69b12ae
Mohammad Saleh
65c1b67a0ad80fd1f69b12af
Simon Baumgartner
claimed_verified
2025-05-20T19:30:17.442Z
6560f697e0a7720b6ae377bc
https://cdn-avatars.huggingface.co/v1/production/uploads/6560f697e0a7720b6ae377bc/8b-p-lQ7KV_VygBJ6Vf3_.jpeg
Simon Baumgartner
user
sens3
65c1b67a0ad80fd1f69b12b0
Jialu Liu
65c1b67a0ad80fd1f69b12b1
Peter J. Liu
claimed_verified
2024-02-07T20:41:08.211Z
62ded294dcdc9043efaa4fc3
/avatars/1294078b8cd5c1901e4021253d877357.svg
Peter J Liu
user
peterjliu
65c1b67a0ad80fd1f69b12b2
Xuanhui Wang
LiPO: Listwise Preference Optimization through Learning-to-Rank
Aligning language models (LMs) with curated human feedback is critical to
control their behaviors in real-world applications. Several recent policy
optimization methods, such as DPO and SLiC, serve as promising alternatives to
the traditional Reinforcement Learning from Human Feedback (RLHF) approach. In
practice, human feedback often comes in a format of a ranked list over multiple
responses to amortize the cost of reading prompt. Multiple responses can also
be ranked by reward models or AI feedback. There lacks such a study on directly
fitting upon a list of responses. In this work, we formulate the LM alignment
as a listwise ranking problem and describe the Listwise Preference Optimization
(LiPO) framework, where the policy can potentially learn more effectively from
a ranked list of plausible responses given the prompt. This view draws an
explicit connection to Learning-to-Rank (LTR), where most existing preference
optimization work can be mapped to existing ranking objectives, especially
pairwise ones. Following this connection, we provide an examination of ranking
objectives that are not well studied for LM alignment withDPO and SLiC as
special cases when list size is two. In particular, we highlight a specific
method, LiPO-{\lambda}, which leverages a state-of-the-art listwise ranking
objective and weights each preference pair in a more advanced manner. We show
that LiPO-{\lambda} can outperform DPO and SLiC by a clear margin on two
preference alignment tasks.
The LiPO framework, which addresses LM alignment as a listwise ranking problem, outperforms DPO and SLiC by effectively learning from ranked lists of responses.
Listwise Preference Optimization LiPO Learning-to-Rank preference optimization listwise ranking objective LiPO-λ