Xwin-LM: Strong and Scalable Alignment Practice for LLMs
In this work, we present Xwin-LM, a comprehensive suite of alignment
methodologies for large language models (LLMs). This suite encompasses several
key techniques, including supervised finetuning (SFT), reward modeling (RM),
rejection sampling finetuning (RS), and direct preference optimization (DPO).
The key components are as follows: (1) Xwin-LM-SFT, models initially finetuned
with high-quality instruction data; (2) Xwin-Pair, a large-scale, multi-turn
preference dataset meticulously annotated using GPT-4; (3) Xwin-RM, reward
models trained on Xwin-Pair, developed at scales of 7B, 13B, and 70B
parameters; (4) Xwin-Set, a multiwise preference dataset in which each prompt
is linked to 64 unique responses generated by Xwin-LM-SFT and scored by
Xwin-RM; (5) Xwin-LM-RS, models finetuned with the highest-scoring responses
from Xwin-Set; (6) Xwin-LM-DPO, models further optimized on Xwin-Set using the
DPO algorithm. Our evaluations on AlpacaEval and MT-bench demonstrate
consistent and significant improvements across the pipeline, demonstrating the
strength and scalability of Xwin-LM. The repository
https://github.com/Xwin-LM/Xwin-LM will be continually updated to foster
community research.
66594d86f1b4f073d80378d2
Bolin Ni
admin_assigned
2024-05-31T09:52:41.072Z
642149592cc2b3c39e7ef6c9
/avatars/fed5d447375cd8c3b2bc9e096fe681de.svg
Bolin Ni
user
nbl97
66594d86f1b4f073d80378d3
JingCheng Hu
admin_assigned
2024-05-31T09:52:47.834Z
625026b7d2d191ac43320c5e
https://cdn-avatars.huggingface.co/v1/production/uploads/625026b7d2d191ac43320c5e/2ExzHlZ-Bk8SQMyBjeY6N.jpeg
Jingcheng Hu
user
reign12
66594d86f1b4f073d80378d4
Yixuan Wei
admin_assigned
2024-05-31T09:53:01.864Z
640efe54a92fedb0e84ee598
/avatars/cb1c6720525bfdbf59a9d3eac93a879e.svg
Yixuan Wei
user
EasonWei
66594d86f1b4f073d80378d5
Houwen Peng
admin_assigned
2024-05-31T09:53:10.851Z
631952b6f18d0b5d999ca397
/avatars/eb87dfc142a7602f8fb888f7b7b60d38.svg
Peng
user
Houwen
66594d86f1b4f073d80378d6
Zheng Zhang
66594d86f1b4f073d80378d7
Gaofeng Meng
66594d86f1b4f073d80378d8
Han Hu
Xwin-LM: Strong and Scalable Alignment Practice for LLMs
In this work, we present Xwin-LM, a comprehensive suite of alignment
methodologies for large language models (LLMs). This suite encompasses several
key techniques, including supervised finetuning (SFT), reward modeling (RM),
rejection sampling finetuning (RS), and direct preference optimization (DPO).
The key components are as follows: (1) Xwin-LM-SFT, models initially finetuned
with high-quality instruction data; (2) Xwin-Pair, a large-scale, multi-turn
preference dataset meticulously annotated using GPT-4; (3) Xwin-RM, reward
models trained on Xwin-Pair, developed at scales of 7B, 13B, and 70B
parameters; (4) Xwin-Set, a multiwise preference dataset in which each prompt
is linked to 64 unique responses generated by Xwin-LM-SFT and scored by
Xwin-RM; (5) Xwin-LM-RS, models finetuned with the highest-scoring responses
from Xwin-Set; (6) Xwin-LM-DPO, models further optimized on Xwin-Set using the
DPO algorithm. Our evaluations on AlpacaEval and MT-bench demonstrate
consistent and significant improvements across the pipeline, demonstrating the
strength and scalability of Xwin-LM. The repository
https://github.com/Xwin-LM/Xwin-LM will be continually updated to foster
community research.
Xwin-LM is a suite of alignment methodologies for large language models that includes supervised finetuning, reward modeling, rejection sampling finetuning, and direct preference optimization, showing consistent improvements on evaluation benchmarks.
supervised finetuning reward modeling rejection sampling finetuning direct preference optimization large language models preference dataset GPT-4 multiwise preference dataset AlpacaEval MT-bench