Soaring from 4K to 400K: Extending LLM's Context with Activation Beacon
The utilization of long contexts poses a big challenge for large language
models due to their limited context window length. Although the context window
can be extended through fine-tuning, it will result in a considerable cost at
both training and inference time, and exert an unfavorable impact to the LLM's
original capabilities. In this work, we propose Activation Beacon, which
condenses LLM's raw activations into more compact forms such that it can
perceive a much longer context with a limited context window. Activation Beacon
is introduced as a plug-and-play module for the LLM. It fully preserves the
LLM's original capability on short contexts while extending the new capability
on processing longer contexts. Besides, it works with short sliding windows to
process the long context, which achieves a competitive memory and time
efficiency in both training and inference. Activation Beacon is learned by the
auto-regression task conditioned on a mixture of beacons with diversified
condensing ratios. Thanks to such a treatment, it can be efficiently trained
purely with short-sequence data in just 10K steps, which consumes less than 9
hours on a single 8xA800 GPU machine. The experimental studies show that
Activation Beacon is able to extend Llama-2-7B's context length by times100
times (from 4K to 400K), meanwhile achieving a superior result on both
long-context generation and understanding tasks. Our model and code will be
available at the BGE repository.
659cc210c80023a02e24bd54
Peitian Zhang
claimed_verified
2024-01-09T09:16:51.563Z
603fb3ebabc4bc39998ff765
/avatars/1a20dec6a1e48017f9a650ac2e510b4e.svg
NaN
user
namespace-Pt
659cc210c80023a02e24bd55
Zheng Liu
claimed_verified
2024-01-25T08:04:55.862Z
64a38c590111d5ff6c3d5f2b
/avatars/ef13dc7ce243819bc0da9b04e778b432.svg
zhengliu
user
zl101
659cc210c80023a02e24bd56
Shitao Xiao
admin_assigned
2024-01-09T08:22:34.399Z
62612679bbcbd1c34f1638af
/avatars/c0675d05a52192ee14e9ab1633353956.svg
Xiao
user
Shitao
659cc210c80023a02e24bd57
Ninglu Shao
admin_assigned
2024-01-09T08:22:44.167Z
653272210917932040597047
/avatars/877a19686dd9780129cfcb85533e2e66.svg
Ninglu Shao
user
rainym00d
659cc210c80023a02e24bd58
Qiwei Ye
admin_assigned
2024-01-09T08:22:51.963Z
62e9de1b4fa4bc6de6c36b7c
https://cdn-avatars.huggingface.co/v1/production/uploads/62e9de1b4fa4bc6de6c36b7c/JbLi6POVPaTSZGcF-9ddQ.jpeg
Qiwei Ye
user
aeros0ul
659cc210c80023a02e24bd59
Zhicheng Dou
claimed_verified
2024-09-23T06:59:35.122Z
66f0bf59e9d50ec57febf751
/avatars/be97941e60064e5dd806c6fe9db3c537.svg
Zhicheng Dou
user
douzc
Soaring from 4K to 400K: Extending LLM's Context with Activation Beacon
The utilization of long contexts poses a big challenge for large language
models due to their limited context window length. Although the context window
can be extended through fine-tuning, it will result in a considerable cost at
both training and inference time, and exert an unfavorable impact to the LLM's
original capabilities. In this work, we propose Activation Beacon, which
condenses LLM's raw activations into more compact forms such that it can
perceive a much longer context with a limited context window. Activation Beacon
is introduced as a plug-and-play module for the LLM. It fully preserves the
LLM's original capability on short contexts while extending the new capability
on processing longer contexts. Besides, it works with short sliding windows to
process the long context, which achieves a competitive memory and time
efficiency in both training and inference. Activation Beacon is learned by the
auto-regression task conditioned on a mixture of beacons with diversified
condensing ratios. Thanks to such a treatment, it can be efficiently trained
purely with short-sequence data in just 10K steps, which consumes less than 9
hours on a single 8xA800 GPU machine. The experimental studies show that
Activation Beacon is able to extend Llama-2-7B's context length by times100
times (from 4K to 400K), meanwhile achieving a superior result on both
long-context generation and understanding tasks. Our model and code will be
available at the BGE repository.
Activation Beacon enhances large language models' ability to process long contexts using compact activation forms, maintaining original capabilities and improving efficiency.
long contexts large language models context window fine-tuning activation beacon raw activations plug-and-play module short contexts short sliding windows memory efficiency time efficiency auto-regression task diversified condensing ratios BGE repository