Accelerating Direct Preference Optimization with Prefix Sharing
Offline paired preference optimization algorithms have become a popular
approach for fine-tuning on preference data, outperforming traditional
supervised fine-tuning in various tasks. However, traditional implementations
often involve redundant computations, especially for tasks with long shared
prompts. We introduce prefix sharing for preference tuning, a novel technique
that processes chosen and rejected responses as one sequence with a shared
prefix. To prevent cross-response contamination, we use a custom block-sparse
attention mask. Our method achieves 1.1-1.5times improvement in training
throughput on popular DPO datasets, without any effect on convergence. When
combined with sequence packing, we observe consistent 1.3-1.6times
speedups, benefiting even datasets with smaller sequence lengths. While we
focus on Direct Preference Optimization (DPO), our approach is applicable to
other paired preference tuning methods. By enhancing computational efficiency,
our work contributes to making preference-based fine-tuning more accessible for
a wider range of applications and model sizes. We open-source our code at
https://github.com/frankxwang/dpo-prefix-sharing.
67226f3f8fedb32c0f117ccc
Franklin Wang
extracted_confirmed
2024-10-30T17:44:39.930Z
6537ec2f33b10aa38f3f6469
/avatars/f25b2dfb2dc4f03b245761dccf3df2d7.svg
Franklin Wang
user
fxwang
67226f3f8fedb32c0f117ccd
Sumanth Hegde
Accelerating Direct Preference Optimization with Prefix Sharing
Offline paired preference optimization algorithms have become a popular
approach for fine-tuning on preference data, outperforming traditional
supervised fine-tuning in various tasks. However, traditional implementations
often involve redundant computations, especially for tasks with long shared
prompts. We introduce prefix sharing for preference tuning, a novel technique
that processes chosen and rejected responses as one sequence with a shared
prefix. To prevent cross-response contamination, we use a custom block-sparse
attention mask. Our method achieves 1.1-1.5times improvement in training
throughput on popular DPO datasets, without any effect on convergence. When
combined with sequence packing, we observe consistent 1.3-1.6times
speedups, benefiting even datasets with smaller sequence lengths. While we
focus on Direct Preference Optimization (DPO), our approach is applicable to
other paired preference tuning methods. By enhancing computational efficiency,
our work contributes to making preference-based fine-tuning more accessible for
a wider range of applications and model sizes. We open-source our code at
https://github.com/frankxwang/dpo-prefix-sharing.
Prefix sharing with a block-sparse attention mask improves training throughput in preference-based fine-tuning without affecting convergence, making it accessible for various model sizes and applications.
offline paired preference optimization fine-tuning Direct Preference Optimization prefix sharing block-sparse attention mask sequence packing