LOGO -- Long cOntext aliGnment via efficient preference Optimization
Long-context models(LCMs) have shown great potential in processing long input
sequences(even more than 100M tokens) conveniently and effectively. With
significant progress, recent research has pointed out that LCMs can accurately
locate token-level salient information within the context. Yet, the generation
performance of these LCMs is far from satisfactory and might result in
misaligned responses, such as hallucinations. To enhance the generation
capability of LCMs, existing works have investigated the effects of data size
and quality for both pre-training and instruction tuning. Though achieving
meaningful improvement, previous methods fall short in either effectiveness or
efficiency. In this paper, we introduce LOGO(Long cOntext aliGnment via
efficient preference Optimization), a training strategy that first introduces
preference optimization for long-context alignment. To overcome the GPU
memory-bound issue caused by the long sequence, LOGO employs a reference-free
preference optimization strategy and adopts a position synthesis method to
construct the training data. By training with only 0.3B data on a single
8timesA800 GPU machine for 16 hours, LOGO allows the Llama-3-8B-Instruct-80K
model to achieve comparable performance with GPT-4 in real-world long-context
tasks while preserving the model's original capabilities on other tasks, e.g.,
language modeling and MMLU. Moreover, LOGO can extend the model's context
window size while enhancing its generation performance.
671b0a2da9a89705ec15a277
Zecheng Tang
admin_assigned
2024-10-25T11:24:53.935Z
64096ef79e9f790c905b846d
https://cdn-avatars.huggingface.co/v1/production/uploads/64096ef79e9f790c905b846d/hVzw656lXdzbCxTtnheud.jpeg
Zecheng Tang
user
ZetangForward
671b0a2da9a89705ec15a278
Zechen Sun
671b0a2da9a89705ec15a279
Juntao Li
admin_assigned
2024-10-25T11:25:12.073Z
6670e285b0c03c4e9d6e0985
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/uCZHm4gKSHZ2b0hpHWgZv.jpeg
Juntao Li
user
douvleplus
671b0a2da9a89705ec15a27a
Qiaoming Zhu
671b0a2da9a89705ec15a27b
Min Zhang
LOGO -- Long cOntext aliGnment via efficient preference Optimization
Long-context models(LCMs) have shown great potential in processing long input
sequences(even more than 100M tokens) conveniently and effectively. With
significant progress, recent research has pointed out that LCMs can accurately
locate token-level salient information within the context. Yet, the generation
performance of these LCMs is far from satisfactory and might result in
misaligned responses, such as hallucinations. To enhance the generation
capability of LCMs, existing works have investigated the effects of data size
and quality for both pre-training and instruction tuning. Though achieving
meaningful improvement, previous methods fall short in either effectiveness or
efficiency. In this paper, we introduce LOGO(Long cOntext aliGnment via
efficient preference Optimization), a training strategy that first introduces
preference optimization for long-context alignment. To overcome the GPU
memory-bound issue caused by the long sequence, LOGO employs a reference-free
preference optimization strategy and adopts a position synthesis method to
construct the training data. By training with only 0.3B data on a single
8timesA800 GPU machine for 16 hours, LOGO allows the Llama-3-8B-Instruct-80K
model to achieve comparable performance with GPT-4 in real-world long-context
tasks while preserving the model's original capabilities on other tasks, e.g.,
language modeling and MMLU. Moreover, LOGO can extend the model's context
window size while enhancing its generation performance.
LOGO, a preference optimization strategy, enhances LCM generation performance and context window size by efficiently training with limited data.
Long-context models token-level salient information preference optimization position synthesis GPU memory-bound issue context window size language modeling MMLU