InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on
  a Single GPU
In modern large language models (LLMs), handling very long context lengths
presents significant challenges as it causes slower inference speeds and
increased memory costs. Additionally, most existing pre-trained LLMs fail to
generalize beyond their original training sequence lengths. To enable efficient
and practical long-context utilization, we introduce InfiniteHiP, a novel, and
practical LLM inference framework that accelerates processing by dynamically
eliminating irrelevant context tokens through a modular hierarchical token
pruning algorithm. Our method also allows generalization to longer sequences by
selectively applying various RoPE adjustment methods according to the internal
attention patterns within LLMs. Furthermore, we offload the key-value cache to
host memory during inference, significantly reducing GPU memory pressure. As a
result, InfiniteHiP enables the processing of up to 3 million tokens on a
single L40s 48GB GPU -- 3x larger -- without any permanent loss of context
information. Our framework achieves an 18.95x speedup in attention decoding for
a 1 million token context without requiring additional training. We implement
our method in the SGLang framework and demonstrate its effectiveness and
practicality through extensive evaluations.
67aebd48225614bbe7f6f271
Heejun Lee
claimed_verified
2025-02-14T08:01:15.423Z
62e622d08e0b2dc6707f8794
/avatars/8c47b5c862f82d4258ba707c932f7f87.svg
Heejun Lee
user
gmlwns5176
67aebd48225614bbe7f6f272
Geon Park
claimed_verified
2025-02-14T08:01:12.988Z
646cae3093badbc8c2e891c7
/avatars/4aae2aca70ea9dc58dd6f9f9b2be15e1.svg
Geon Park
user
geonp
67aebd48225614bbe7f6f273
Jaduk Suh
admin_assigned
2025-02-14T12:00:46.888Z
657ffd9ba2ef32167533f04a
/avatars/e180e063c810c15d02b494727e962b84.svg
Jaduk Suh
user
Losif63
67aebd48225614bbe7f6f274
Sung Ju Hwang
InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on
  a Single GPU
In modern large language models (LLMs), handling very long context lengths
presents significant challenges as it causes slower inference speeds and
increased memory costs. Additionally, most existing pre-trained LLMs fail to
generalize beyond their original training sequence lengths. To enable efficient
and practical long-context utilization, we introduce InfiniteHiP, a novel, and
practical LLM inference framework that accelerates processing by dynamically
eliminating irrelevant context tokens through a modular hierarchical token
pruning algorithm. Our method also allows generalization to longer sequences by
selectively applying various RoPE adjustment methods according to the internal
attention patterns within LLMs. Furthermore, we offload the key-value cache to
host memory during inference, significantly reducing GPU memory pressure. As a
result, InfiniteHiP enables the processing of up to 3 million tokens on a
single L40s 48GB GPU -- 3x larger -- without any permanent loss of context
information. Our framework achieves an 18.95x speedup in attention decoding for
a 1 million token context without requiring additional training. We implement
our method in the SGLang framework and demonstrate its effectiveness and
practicality through extensive evaluations.
InfiniteHiP is a novel LLM inference framework that enhances long-context processing with token pruning, RoPE adjustment, and host memory offloading, enabling faster and more memory-efficient operation.
LLMs InfiniteHiP token pruning hierarchical token pruning algorithm RoPE adjustment key-value cache host memory GPU memory pressure