LongPO: Long Context Self-Evolution of Large Language Models through
  Short-to-Long Preference Optimization
Large Language Models (LLMs) have demonstrated remarkable capabilities
through pretraining and alignment. However, superior short-context LLMs may
underperform in long-context scenarios due to insufficient long-context
alignment. This alignment process remains challenging due to the impracticality
of human annotation for extended contexts and the difficulty in balancing
short- and long-context performance. To address these challenges, we introduce
LongPO, that enables short-context LLMs to self-evolve to excel on long-context
tasks by internally transferring short-context capabilities. LongPO harnesses
LLMs to learn from self-generated short-to-long preference data, comprising
paired responses generated for identical instructions with long-context inputs
and their compressed short-context counterparts, respectively. This preference
reveals capabilities and potentials of LLMs cultivated during short-context
alignment that may be diminished in under-aligned long-context scenarios.
Additionally, LongPO incorporates a short-to-long KL constraint to mitigate
short-context performance decline during long-context alignment. When applied
to Mistral-7B-Instruct-v0.2 from 128K to 512K context lengths, LongPO fully
retains short-context performance and largely outperforms naive SFT and DPO in
both long- and short-context tasks. Specifically, \ourMethod-trained models can
achieve results on long-context benchmarks comparable to, or even surpassing,
those of superior LLMs (e.g., GPT-4-128K) that involve extensive long-context
annotation and larger parameter scales.
67b6948dbef24bad725b5d4b
Guanzheng Chen
admin_assigned
2025-02-20T16:08:30.816Z
645475e2548f22be59847604
https://cdn-avatars.huggingface.co/v1/production/uploads/645475e2548f22be59847604/EhSurrZ25u31qQ2TVXQXt.jpeg
Chen
user
Guanzheng
67b6948dbef24bad725b5d4c
Xin Li
67b6948dbef24bad725b5d4d
Michael Qizhe Shieh
67b6948dbef24bad725b5d4e
Lidong Bing
admin_assigned
2025-02-20T16:08:48.204Z
6454685a548f22be598414c4
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eMjMWKJ-AouF7eY1-RzGF.jpeg
Lidong Bing
user
LidongBing
LongPO: Long Context Self-Evolution of Large Language Models through
  Short-to-Long Preference Optimization
Large Language Models (LLMs) have demonstrated remarkable capabilities
through pretraining and alignment. However, superior short-context LLMs may
underperform in long-context scenarios due to insufficient long-context
alignment. This alignment process remains challenging due to the impracticality
of human annotation for extended contexts and the difficulty in balancing
short- and long-context performance. To address these challenges, we introduce
LongPO, that enables short-context LLMs to self-evolve to excel on long-context
tasks by internally transferring short-context capabilities. LongPO harnesses
LLMs to learn from self-generated short-to-long preference data, comprising
paired responses generated for identical instructions with long-context inputs
and their compressed short-context counterparts, respectively. This preference
reveals capabilities and potentials of LLMs cultivated during short-context
alignment that may be diminished in under-aligned long-context scenarios.
Additionally, LongPO incorporates a short-to-long KL constraint to mitigate
short-context performance decline during long-context alignment. When applied
to Mistral-7B-Instruct-v0.2 from 128K to 512K context lengths, LongPO fully
retains short-context performance and largely outperforms naive SFT and DPO in
both long- and short-context tasks. Specifically, \ourMethod-trained models can
achieve results on long-context benchmarks comparable to, or even surpassing,
those of superior LLMs (e.g., GPT-4-128K) that involve extensive long-context
annotation and larger parameter scales.
LongPO enables short-context LLMs to adapt to long-context tasks by self-learning from preference data and maintaining short-context performance.
LongPO LLMs short-context long-context self-evolve self-generated short-to-long preference data KL constraint SFT DPO context lengths long-context benchmarks GPT-4-128K