SepLLM: Accelerate Large Language Models by Compressing One Segment into
  One Separator
Large Language Models (LLMs) have exhibited exceptional performance across a
spectrum of natural language processing tasks. However, their substantial sizes
pose considerable challenges, particularly in computational demands and
inference speed, due to their quadratic complexity. In this work, we have
identified a key pattern: certain seemingly meaningless special tokens (i.e.,
separators) contribute disproportionately to attention scores compared to
semantically meaningful tokens. This observation suggests that information of
the segments between these separator tokens can be effectively condensed into
the separator tokens themselves without significant information loss. Guided by
this insight, we introduce SepLLM, a plug-and-play framework that accelerates
inference by compressing these segments and eliminating redundant tokens.
Additionally, we implement efficient kernels for training acceleration.
Experimental results across training-free, training-from-scratch, and
post-training settings demonstrate SepLLM's effectiveness. Notably, using the
Llama-3-8B backbone, SepLLM achieves over 50% reduction in KV cache on the
GSM8K-CoT benchmark while maintaining comparable performance. Furthermore, in
streaming settings, SepLLM effectively processes sequences of up to 4 million
tokens or more while maintaining consistent language modeling capabilities.
67615d54402071c9e07c0c42
Guoxuan Chen
67615d54402071c9e07c0c43
Han Shi
claimed_verified
2024-12-17T11:18:01.539Z
637751ad7169ed981703d26a
/avatars/9823f115040a3d6b0240038010e46cb6.svg
Han Shi
user
shihan96
67615d54402071c9e07c0c44
Jiawei Li
67615d54402071c9e07c0c45
Yihang Gao
67615d54402071c9e07c0c46
Xiaozhe Ren
67615d54402071c9e07c0c47
Yimeng Chen
claimed_verified
2025-07-22T13:22:25.543Z
6836f97bed0b13d776f1006a
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/-h4S4OJYJG02PFkYXKPYX.png
Yimeng Chen
user
YimengChen
67615d54402071c9e07c0c48
Xin Jiang
67615d54402071c9e07c0c49
Zhenguo Li
67615d54402071c9e07c0c4a
Weiyang Liu
67615d54402071c9e07c0c4b
Chao Huang
SepLLM: Accelerate Large Language Models by Compressing One Segment into
  One Separator
Large Language Models (LLMs) have exhibited exceptional performance across a
spectrum of natural language processing tasks. However, their substantial sizes
pose considerable challenges, particularly in computational demands and
inference speed, due to their quadratic complexity. In this work, we have
identified a key pattern: certain seemingly meaningless special tokens (i.e.,
separators) contribute disproportionately to attention scores compared to
semantically meaningful tokens. This observation suggests that information of
the segments between these separator tokens can be effectively condensed into
the separator tokens themselves without significant information loss. Guided by
this insight, we introduce SepLLM, a plug-and-play framework that accelerates
inference by compressing these segments and eliminating redundant tokens.
Additionally, we implement efficient kernels for training acceleration.
Experimental results across training-free, training-from-scratch, and
post-training settings demonstrate SepLLM's effectiveness. Notably, using the
Llama-3-8B backbone, SepLLM achieves over 50% reduction in KV cache on the
GSM8K-CoT benchmark while maintaining comparable performance. Furthermore, in
streaming settings, SepLLM effectively processes sequences of up to 4 million
tokens or more while maintaining consistent language modeling capabilities.
SepLLM compresses segments between separator tokens to accelerate inference and training of large language models with minimal performance loss.
Large Language Models LLMs attention scores separator tokens SegLLM KV cache GSM8K-CoT benchmark