LLoCO: Learning Long Contexts Offline
Processing long contexts remains a challenge for large language models (LLMs)
due to the quadratic computational and memory overhead of the self-attention
mechanism and the substantial KV cache sizes during generation. We propose a
novel approach to address this problem by learning contexts offline through
context compression and in-domain parameter-efficient finetuning. Our method
enables an LLM to create a concise representation of the original context and
efficiently retrieve relevant information to answer questions accurately. We
introduce LLoCO, a technique that combines context compression, retrieval, and
parameter-efficient finetuning using LoRA. Our approach extends the effective
context window of a 4k token LLaMA2-7B model to handle up to 128k tokens. We
evaluate our approach on several long-context question-answering datasets,
demonstrating that LLoCO significantly outperforms in-context learning while
using 30times fewer tokens during inference. LLoCO achieves up to
7.62times speed-up and substantially reduces the cost of long document
question answering, making it a promising solution for efficient long context
processing. Our code is publicly available at
https://github.com/jeffreysijuntan/lloco.
661896d0de161bdcb0f59be4
Sijun Tan
admin_assigned
2024-04-18T12:30:36.653Z
652867b09903f7a1c9f7cbf1
/avatars/1d1d3665328943ccdbd6a25ce03f2f10.svg
Sijun Tan
user
sijuntan
661896d0de161bdcb0f59be5
Xiuyu Li
claimed_verified
2024-04-12T03:05:21.524Z
644570ba2d91b15b4c7f6311
/avatars/d5e66012066d0c330b8f23718b1499d8.svg
Xiuyu Li
user
xiuyul
661896d0de161bdcb0f59be6
Shishir Patil
admin_assigned
2024-04-18T12:30:43.149Z
6471b9f6094820190c324eec
https://cdn-avatars.huggingface.co/v1/production/uploads/6471b9f6094820190c324eec/yb62BiXzRFY1_pS0GAMGq.jpeg
Shishir Patil
user
shishirpatil
661896d0de161bdcb0f59be7
Ziyang Wu
admin_assigned
2024-04-18T12:31:39.967Z
648fbaff4852c28c4cff7a3b
/avatars/f987bb904580865676c8541839dd7615.svg
wuziyang
user
wzyohyes
661896d0de161bdcb0f59be8
Tianjun Zhang
admin_assigned
2024-04-18T12:31:47.275Z
6374cd6b6ea8da14f8fef8dc
https://cdn-avatars.huggingface.co/v1/production/uploads/6374cd6b6ea8da14f8fef8dc/l13bg0tKDjCnUw3I895QZ.png
Tianjun Zhang
user
tianjunz
661896d0de161bdcb0f59be9
Kurt Keutzer
admin_assigned
2024-04-18T12:32:02.210Z
6251bf4b183aa4266924ad91
https://cdn-avatars.huggingface.co/v1/production/uploads/1678041834400-6251bf4b183aa4266924ad91.jpeg
Kurt Keutzer
user
kurtkeutzer
661896d0de161bdcb0f59bea
Joseph E. Gonzalez
admin_assigned
2024-04-18T12:32:09.861Z
645d2e8401f4eaab2a0878ce
/avatars/1273c5fb607b4b622a746a42692fa632.svg
Joseph E. Gonzalez
user
ProfJoeyG
661896d0de161bdcb0f59beb
Raluca Ada Popa
LLoCO: Learning Long Contexts Offline
Processing long contexts remains a challenge for large language models (LLMs)
due to the quadratic computational and memory overhead of the self-attention
mechanism and the substantial KV cache sizes during generation. We propose a
novel approach to address this problem by learning contexts offline through
context compression and in-domain parameter-efficient finetuning. Our method
enables an LLM to create a concise representation of the original context and
efficiently retrieve relevant information to answer questions accurately. We
introduce LLoCO, a technique that combines context compression, retrieval, and
parameter-efficient finetuning using LoRA. Our approach extends the effective
context window of a 4k token LLaMA2-7B model to handle up to 128k tokens. We
evaluate our approach on several long-context question-answering datasets,
demonstrating that LLoCO significantly outperforms in-context learning while
using 30times fewer tokens during inference. LLoCO achieves up to
7.62times speed-up and substantially reduces the cost of long document
question answering, making it a promising solution for efficient long context
processing. Our code is publicly available at
https://github.com/jeffreysijuntan/lloco.
LLoCO method enhances large language models' ability to handle long contexts efficiently by combining context compression, retrieval, and parameter-efficient finetuning.
self-attention context compression parameter-efficient finetuning LoRA LLaMA2-7B long-context question-answering in-context learning