PopAlign: Diversifying Contrasting Patterns for a More Comprehensive
  Alignment
Alignment of large language models (LLMs) involves training models on
preference-contrastive output pairs to adjust their responses according to
human preferences. To obtain such contrastive pairs, traditional methods like
RLHF and RLAIF rely on limited contrasting patterns, such as varying model
variants or decoding temperatures. This singularity leads to two issues: (1)
alignment is not comprehensive; and thereby (2) models are susceptible to
jailbreaking attacks. To address these issues, we investigate how to construct
more comprehensive and diversified contrasting patterns to enhance preference
data (RQ1) and verify the impact of the diversification of contrasting patterns
on model alignment (RQ2). For RQ1, we propose PopAlign, a framework that
integrates diversified contrasting patterns across the prompt, model, and
pipeline levels, introducing six contrasting strategies that do not require
additional feedback labeling procedures. Regarding RQ2, we conduct thorough
experiments demonstrating that PopAlign significantly outperforms existing
methods, leading to more comprehensive alignment.
6711cdb07da5c0a86f649718
Zekun Moore Wang
admin_assigned
2024-10-18T12:37:40.404Z
6149a9e95347647e6bb68882
https://cdn-avatars.huggingface.co/v1/production/uploads/6149a9e95347647e6bb68882/Jddln1FxScCeVgTSCNBpr.png
Zekun Moore Wang
user
ZenMoore
6711cdb07da5c0a86f649719
Shawn Wang
6711cdb07da5c0a86f64971a
Kang Zhu
claimed_verified
2024-10-18T08:55:13.210Z
6578265ddea7e2122d02f6ba
https://cdn-avatars.huggingface.co/v1/production/uploads/6578265ddea7e2122d02f6ba/Bh6JjoVF5ceLSjV7Z7nTk.jpeg
kang zhu
user
kangz
6711cdb07da5c0a86f64971b
Jiaheng Liu
admin_assigned
2024-10-18T12:38:22.673Z
65377c30e48353201e6fdda0
/avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
Jiaheng Liu
user
CheeryLJH
6711cdb07da5c0a86f64971c
Ke Xu
6711cdb07da5c0a86f64971d
Jie Fu
6711cdb07da5c0a86f64971e
Wangchunshu Zhou
admin_assigned
2024-10-18T12:38:57.876Z
628c8598ef14f971b698107f
/avatars/3a4ad87e6b5f9e836a1160d869df1447.svg
Zhou
user
Wangchunshu
6711cdb07da5c0a86f64971f
Wenhao Huang
admin_assigned
2024-10-18T12:39:13.119Z
641e5bf65f274a0a92c2f6a2
/avatars/c15a54c51998c0e6367685e8e1737ec9.svg
Wenhao Huang
user
EZ-hwh
PopAlign: Diversifying Contrasting Patterns for a More Comprehensive
  Alignment
Alignment of large language models (LLMs) involves training models on
preference-contrastive output pairs to adjust their responses according to
human preferences. To obtain such contrastive pairs, traditional methods like
RLHF and RLAIF rely on limited contrasting patterns, such as varying model
variants or decoding temperatures. This singularity leads to two issues: (1)
alignment is not comprehensive; and thereby (2) models are susceptible to
jailbreaking attacks. To address these issues, we investigate how to construct
more comprehensive and diversified contrasting patterns to enhance preference
data (RQ1) and verify the impact of the diversification of contrasting patterns
on model alignment (RQ2). For RQ1, we propose PopAlign, a framework that
integrates diversified contrasting patterns across the prompt, model, and
pipeline levels, introducing six contrasting strategies that do not require
additional feedback labeling procedures. Regarding RQ2, we conduct thorough
experiments demonstrating that PopAlign significantly outperforms existing
methods, leading to more comprehensive alignment.
PopAlign enhances large language model alignment by introducing diverse contrasting patterns across prompts, models, and pipelines, leading to improved alignment compared to traditional methods.
large language models LLMs alignment preference-contrastive output pairs RLHF RLAIF contrasting patterns jailbreaking attacks PopAlign comprehensive alignment