LongRecipe: Recipe for Efficient Long Context Generalization in Large
  Languge Models
Large language models (LLMs) face significant challenges in handling
long-context tasks because of their limited effective context window size
during pretraining, which restricts their ability to generalize over extended
sequences. Meanwhile, extending the context window in LLMs through
post-pretraining is highly resource-intensive. To address this, we introduce
**LongRecipe**, an efficient training strategy for extending the context window
of LLMs, including impactful token analysis, position index transformation, and
training optimization strategies. It simulates long-sequence inputs while
maintaining training efficiency and significantly improves the model's
understanding of long-range dependencies. Experiments on three types of LLMs
show that LongRecipe can utilize long sequences while requiring only 30% of the
target context window size, and reduces computational training resource over
85% compared to full sequence training. Furthermore, LongRecipe also preserves
the original LLM's capabilities in general tasks. Ultimately, *we can extend
the effective context window of open-source LLMs from 8k to 128k, achieving
performance close to GPT-4 with just one day of dedicated training using a
single GPU with 80G memory.* Our code is released at the
[link](https://github.com/zhiyuanhubj/LongRecipe).
66d7dd20b5c8cfdee7d38006
Zhiyuan Hu
claimed_verified
2024-09-04T08:33:49.067Z
64351475901c5734bcb64248
/avatars/12346d4301c1bfb00ce0ea128a93cc15.svg
Zhiyuan Hu
user
zhiyuanhucs
66d7dd20b5c8cfdee7d38007
Yuliang Liu
claimed_verified
2024-09-04T07:15:02.277Z
6529f79e802e3d1a4f8ec662
/avatars/d05320c370a6497d8792ef5acb563dd5.svg
Yuliang Liu
user
yuliang03181
66d7dd20b5c8cfdee7d38008
Jinman Zhao
claimed_verified
2024-09-04T16:21:03.953Z
6546ecc9cd0a56213929fd18
/avatars/9f4d4e369f0ca6e5669b038badc4e622.svg
jinman
user
zhaojinm
66d7dd20b5c8cfdee7d38009
Suyuchen Wang
claimed_verified
2024-09-04T07:15:03.844Z
62bb1e0f3ff437e49a3088e5
https://cdn-avatars.huggingface.co/v1/production/uploads/62bb1e0f3ff437e49a3088e5/MWNanci3x5g780xh-704U.png
Suyuchen Wang
user
sheryc
66d7dd20b5c8cfdee7d3800a
Yan Wang
66d7dd20b5c8cfdee7d3800b
Wei Shen
claimed_verified
2024-09-12T13:30:29.260Z
6468823272d9180d4ac90bdf
/avatars/70cb7d65d30ecb944595000ceeeedb1b.svg
Wei Shen
user
Swtheking
66d7dd20b5c8cfdee7d3800c
Qing Gu
66d7dd20b5c8cfdee7d3800d
Anh Tuan Luu
admin_assigned
2024-09-04T08:16:02.258Z
655722e80438e0854fae7554
/avatars/b93a74f7c7880f9fe0f3ffb47e2aef5e.svg
Luu Anh Tuan
user
anhtuanluu36
66d7dd20b5c8cfdee7d3800e
See-Kiong Ng
66d7dd20b5c8cfdee7d3800f
Zhiwei Jiang
admin_assigned
2024-09-04T08:15:49.639Z
66446066b5a21591719f3f8e
/avatars/923e7786da7bdaa69c7e3d1567a6a156.svg
Jiang Zhiwei
user
Laziobird
66d7dd20b5c8cfdee7d38010
Bryan Hooi
admin_assigned
2024-09-04T08:14:39.419Z
651d8032c50012d33e914f2f
/avatars/0a44c9f51fc50ce86582e328c361ea00.svg
Bryan Hooi
user
bhooi
LongRecipe: Recipe for Efficient Long Context Generalization in Large
  Languge Models
Large language models (LLMs) face significant challenges in handling
long-context tasks because of their limited effective context window size
during pretraining, which restricts their ability to generalize over extended
sequences. Meanwhile, extending the context window in LLMs through
post-pretraining is highly resource-intensive. To address this, we introduce
**LongRecipe**, an efficient training strategy for extending the context window
of LLMs, including impactful token analysis, position index transformation, and
training optimization strategies. It simulates long-sequence inputs while
maintaining training efficiency and significantly improves the model's
understanding of long-range dependencies. Experiments on three types of LLMs
show that LongRecipe can utilize long sequences while requiring only 30% of the
target context window size, and reduces computational training resource over
85% compared to full sequence training. Furthermore, LongRecipe also preserves
the original LLM's capabilities in general tasks. Ultimately, *we can extend
the effective context window of open-source LLMs from 8k to 128k, achieving
performance close to GPT-4 with just one day of dedicated training using a
single GPU with 80G memory.* Our code is released at the
[link](https://github.com/zhiyuanhubj/LongRecipe).
LongRecipe extends the context window of large language models efficiently with reduced resources and preserved performance, achieving a 16-fold increase from 8k to 128k context size.
large language models (LLMs) context window pretraining post-pretraining token analysis position index transformation training optimization strategies long-sequence inputs long-range dependencies GPT-4