LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models
We present LongLoRA, an efficient fine-tuning approach that extends the
context sizes of pre-trained large language models (LLMs), with limited
computation cost. Typically, training LLMs with long context sizes is
computationally expensive, requiring extensive training hours and GPU
resources. For example, training on the context length of 8192 needs 16x
computational costs in self-attention layers as that of 2048. In this paper, we
speed up the context extension of LLMs in two aspects. On the one hand,
although dense global attention is needed during inference, fine-tuning the
model can be effectively and efficiently done by sparse local attention. The
proposed shift short attention effectively enables context extension, leading
to non-trivial computation saving with similar performance to fine-tuning with
vanilla attention. Particularly, it can be implemented with only two lines of
code in training, while being optional in inference. On the other hand, we
revisit the parameter-efficient fine-tuning regime for context expansion.
Notably, we find that LoRA for context extension works well under the premise
of trainable embedding and normalization. LongLoRA demonstrates strong
empirical results on various tasks on LLaMA2 models from 7B/13B to 70B.
LongLoRA adopts LLaMA2 7B from 4k context to 100k, or LLaMA2 70B to 32k on a
single 8x A100 machine. LongLoRA extends models' context while retaining their
original architectures, and is compatible with most existing techniques, like
FlashAttention-2. In addition, to make LongLoRA practical, we collect a
dataset, LongQA, for supervised fine-tuning. It contains more than 3k long
context question-answer pairs.
650ce7f6cbd0c7d550c64d9c
Yukang Chen
admin_assigned
2023-09-22T14:54:21.276Z
62919485a29097b211bc7b83
https://cdn-avatars.huggingface.co/v1/production/uploads/1653710384819-62919485a29097b211bc7b83.png
YukangChen
user
Yukang
650ce7f6cbd0c7d550c64d9d
Shengju Qian
admin_assigned
2023-09-22T14:54:29.002Z
6380580f42cedbc20c7bef71
/avatars/8d710e0de551cd2bf545cc31fcaf099d.svg
Shengju Qian
user
thesouthfrog
650ce7f6cbd0c7d550c64d9e
Haotian Tang
650ce7f6cbd0c7d550c64d9f
Xin Lai
admin_assigned
2023-09-22T14:54:47.313Z
642bddc1fc41757877f68327
/avatars/f275237f36a112624d59a7e3f73237d3.svg
Xin Lai
user
xinlai
650ce7f6cbd0c7d550c64da0
Zhijian Liu
claimed_verified
2023-09-22T15:32:56.869Z
650dac79b959b0e1d41d7378
https://cdn-avatars.huggingface.co/v1/production/uploads/650dac79b959b0e1d41d7378/mzbN0MFk3k8b94FQ40I7L.jpeg
Zhijian Liu
user
zhijianliu
650ce7f6cbd0c7d550c64da1
Song Han
admin_assigned
2023-09-22T14:55:46.154Z
63797f727df2fefdcaf3ff7e
https://cdn-avatars.huggingface.co/v1/production/uploads/1668906853549-noauth.jpeg
Song
user
songhan
650ce7f6cbd0c7d550c64da2
Jiaya Jia
LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models
We present LongLoRA, an efficient fine-tuning approach that extends the
context sizes of pre-trained large language models (LLMs), with limited
computation cost. Typically, training LLMs with long context sizes is
computationally expensive, requiring extensive training hours and GPU
resources. For example, training on the context length of 8192 needs 16x
computational costs in self-attention layers as that of 2048. In this paper, we
speed up the context extension of LLMs in two aspects. On the one hand,
although dense global attention is needed during inference, fine-tuning the
model can be effectively and efficiently done by sparse local attention. The
proposed shift short attention effectively enables context extension, leading
to non-trivial computation saving with similar performance to fine-tuning with
vanilla attention. Particularly, it can be implemented with only two lines of
code in training, while being optional in inference. On the other hand, we
revisit the parameter-efficient fine-tuning regime for context expansion.
Notably, we find that LoRA for context extension works well under the premise
of trainable embedding and normalization. LongLoRA demonstrates strong
empirical results on various tasks on LLaMA2 models from 7B/13B to 70B.
LongLoRA adopts LLaMA2 7B from 4k context to 100k, or LLaMA2 70B to 32k on a
single 8x A100 machine. LongLoRA extends models' context while retaining their
original architectures, and is compatible with most existing techniques, like
FlashAttention-2. In addition, to make LongLoRA practical, we collect a
dataset, LongQA, for supervised fine-tuning. It contains more than 3k long
context question-answer pairs.
LongLoRA extends LLM context sizes efficiently using sparse local attention and parameter-efficient fine-tuning, demonstrating strong results with minimal computational cost.
LongLoRA dense global attention sparse local attention shift short attention parameter-efficient fine-tuning LoRA trainable embedding normalization LongQA