sDPO: Don't Use Your Data All at Once
As development of large language models (LLM) progresses, aligning them with
human preferences has become increasingly important. We propose stepwise DPO
(sDPO), an extension of the recently popularized direct preference optimization
(DPO) for alignment tuning. This approach involves dividing the available
preference datasets and utilizing them in a stepwise manner, rather than
employing it all at once. We demonstrate that this method facilitates the use
of more precisely aligned reference models within the DPO training framework.
Furthermore, sDPO trains the final model to be more performant, even
outperforming other popular LLMs with more parameters.
6606447bf631d41ec45b1ac0
Dahyun Kim
6606447bf631d41ec45b1ac1
Yungi Kim
6606447bf631d41ec45b1ac2
Wonho Song
6606447bf631d41ec45b1ac3
Hyeonwoo Kim
6606447bf631d41ec45b1ac4
Yunsu Kim
6606447bf631d41ec45b1ac5
Sanghoon Kim
6606447bf631d41ec45b1ac6
Chanjun Park
claimed_verified
2024-04-09T19:21:22.718Z
64b9fd6910430817357cd9b2
https://cdn-avatars.huggingface.co/v1/production/uploads/64b9fd6910430817357cd9b2/Zm_Ngp20IlHlLIOFlNXPo.jpeg
Chanjun Park
user
Chanjun
sDPO: Don't Use Your Data All at Once
As development of large language models (LLM) progresses, aligning them with
human preferences has become increasingly important. We propose stepwise DPO
(sDPO), an extension of the recently popularized direct preference optimization
(DPO) for alignment tuning. This approach involves dividing the available
preference datasets and utilizing them in a stepwise manner, rather than
employing it all at once. We demonstrate that this method facilitates the use
of more precisely aligned reference models within the DPO training framework.
Furthermore, sDPO trains the final model to be more performant, even
outperforming other popular LLMs with more parameters.
A stepwise direct preference optimization approach improves the alignment of large language models with human preferences and enhances their performance.
large language models LLM direct preference optimization DPO stepwise DPO sDPO preference datasets reference models