LongAlign: A Recipe for Long Context Alignment of Large Language Models
Extending large language models to effectively handle long contexts requires
instruction fine-tuning on input sequences of similar length. To address this,
we present LongAlign -- a recipe of the instruction data, training, and
evaluation for long context alignment. First, we construct a long
instruction-following dataset using Self-Instruct. To ensure the data
diversity, it covers a broad range of tasks from various long context sources.
Second, we adopt the packing and sorted batching strategies to speed up
supervised fine-tuning on data with varied length distributions. Additionally,
we develop a loss weighting method to balance the contribution to the loss
across different sequences during packing training. Third, we introduce the
LongBench-Chat benchmark for evaluating instruction-following capabilities on
queries of 10k-100k in length. Experiments show that LongAlign outperforms
existing recipes for LLMs in long context tasks by up to 30\%, while also
maintaining their proficiency in handling short, generic tasks. The code, data,
and long-aligned models are open-sourced at https://github.com/THUDM/LongAlign.
65bb14cabd9e29f771f5d0cd
Yushi Bai
claimed_verified
2024-02-01T08:44:19.478Z
64ed568ccf6118a9379a61b8
/avatars/6d040cbcb4a9b624cbe64c9d01cd5c88.svg
Yushi Bai
user
bys0318
65bb14cabd9e29f771f5d0ce
Xin Lv
admin_assigned
2024-02-01T09:42:35.924Z
648c64829c935db2b527a764
https://cdn-avatars.huggingface.co/v1/production/uploads/648c64829c935db2b527a764/AnTu1dvpac3hUxLQo8A8K.jpeg
Xin Lv
user
davidlvxin
65bb14cabd9e29f771f5d0cf
Jiajie Zhang
claimed_verified
2024-09-04T07:26:17.362Z
66cdd285c51a915bd5f2d017
/avatars/14e5794307e4672b1b51d26b31227e0f.svg
Jiajie Zhang
user
NeoZ123
65bb14cabd9e29f771f5d0d0
Yuze He
admin_assigned
2024-02-01T09:43:10.123Z
64c903957b4d0d947ce86bc6
/avatars/61d70a3ba00c83a5950f5c909a1a06f8.svg
Yuze He
user
hyz317
65bb14cabd9e29f771f5d0d1
Ji Qi
claimed_verified
2025-07-02T08:24:26.488Z
6864b9f90a269bace8c92164
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/zSeWHZRCZtzZ13CzFrr5u.png
Ji Qi
user
miracle11121
65bb14cabd9e29f771f5d0d2
Lei Hou
65bb14cabd9e29f771f5d0d3
Jie Tang
admin_assigned
2024-02-01T09:44:23.312Z
640dff05474aa6f89556677e
/avatars/1b4591c7322d649c797b3125148f1915.svg
Jie Tang
user
jerytang
65bb14cabd9e29f771f5d0d4
Yuxiao Dong
admin_assigned
2024-02-01T09:44:29.296Z
640e73bdfdeaae1390857b62
/avatars/cd6779e30f716002a7838ed93d5c0754.svg
Yuxiao Dong
user
yuxiaod
65bb14cabd9e29f771f5d0d5
Juanzi Li
LongAlign: A Recipe for Long Context Alignment of Large Language Models
Extending large language models to effectively handle long contexts requires
instruction fine-tuning on input sequences of similar length. To address this,
we present LongAlign -- a recipe of the instruction data, training, and
evaluation for long context alignment. First, we construct a long
instruction-following dataset using Self-Instruct. To ensure the data
diversity, it covers a broad range of tasks from various long context sources.
Second, we adopt the packing and sorted batching strategies to speed up
supervised fine-tuning on data with varied length distributions. Additionally,
we develop a loss weighting method to balance the contribution to the loss
across different sequences during packing training. Third, we introduce the
LongBench-Chat benchmark for evaluating instruction-following capabilities on
queries of 10k-100k in length. Experiments show that LongAlign outperforms
existing recipes for LLMs in long context tasks by up to 30\%, while also
maintaining their proficiency in handling short, generic tasks. The code, data,
and long-aligned models are open-sourced at https://github.com/THUDM/LongAlign.
LongAlign improves the handling of long contexts in large language models through specialized instruction datasets, packing and batching strategies, loss weighting, and a benchmark for extended queries.
Self-Instruct long instruction-following dataset packing strategies sorted batching loss weighting LongBench-Chat benchmark long context alignment