HoliTom: Holistic Token Merging for Fast Video Large Language Models
Video large language models (video LLMs) excel at video comprehension but
face significant computational inefficiency due to redundant video tokens.
Existing token pruning methods offer solutions. However, approaches operating
within the LLM (inner-LLM pruning), such as FastV, incur intrinsic
computational overhead in shallow layers. In contrast, methods performing token
pruning before the LLM (outer-LLM pruning) primarily address spatial redundancy
within individual frames or limited temporal windows, neglecting the crucial
global temporal dynamics and correlations across longer video sequences. This
leads to sub-optimal spatio-temporal reduction and does not leverage video
compressibility fully. Crucially, the synergistic potential and mutual
influence of combining these strategies remain unexplored. To further reduce
redundancy, we introduce HoliTom, a novel training-free holistic token merging
framework. HoliTom employs outer-LLM pruning through global redundancy-aware
temporal segmentation, followed by spatial-temporal merging to reduce visual
tokens by over 90%, significantly alleviating the LLM's computational burden.
Complementing this, we introduce a robust inner-LLM token similarity-based
merging approach, designed for superior performance and compatibility with
outer-LLM pruning. Evaluations demonstrate our method's promising
efficiency-performance trade-off on LLaVA-OneVision-7B, reducing computational
costs to 6.9% of FLOPs while maintaining 99.1% of the original performance.
Furthermore, we achieve a 2.28x reduction in Time-To-First-Token (TTFT) and a
1.32x acceleration in decoding throughput, highlighting the practical benefits
of our integrated pruning approach for efficient video LLMs inference.
6836a5a9bec1d6dbb3e10454
Kele Shao
claimed_verified
2025-05-28T08:56:19.978Z
6696755fd26a65bd255184d3
/avatars/8d46c21a7b23f0100a7e3385fea61edf.svg
Kele Shao (SII)
user
keleshao
6836a5a9bec1d6dbb3e10455
Keda Tao
claimed_verified
2025-07-29T07:11:13.820Z
65ce28c6340c3e914285aa58
/avatars/ffaa6d6ce92274bff960f8ea229a37f8.svg
Keda TAO
user
KD-TAO
6836a5a9bec1d6dbb3e10456
Can Qin
6836a5a9bec1d6dbb3e10457
Haoxuan You
6836a5a9bec1d6dbb3e10458
Yang Sui
claimed_verified
2025-05-29T20:18:55.251Z
63787b13500186f250ba377c
/avatars/96bb6051662109e9cd25e6df7d738f17.svg
yangsui
user
yangsui
6836a5a9bec1d6dbb3e10459
Huan Wang
claimed_verified
2025-05-28T10:11:24.697Z
62b624f3b52bef716e248fd7
https://cdn-avatars.huggingface.co/v1/production/uploads/62b624f3b52bef716e248fd7/AllcccKH-eBWduA8KVnOQ.png
Huan Wang
user
Huan-WhoRegisteredMyName
HoliTom: Holistic Token Merging for Fast Video Large Language Models
Video large language models (video LLMs) excel at video comprehension but
face significant computational inefficiency due to redundant video tokens.
Existing token pruning methods offer solutions. However, approaches operating
within the LLM (inner-LLM pruning), such as FastV, incur intrinsic
computational overhead in shallow layers. In contrast, methods performing token
pruning before the LLM (outer-LLM pruning) primarily address spatial redundancy
within individual frames or limited temporal windows, neglecting the crucial
global temporal dynamics and correlations across longer video sequences. This
leads to sub-optimal spatio-temporal reduction and does not leverage video
compressibility fully. Crucially, the synergistic potential and mutual
influence of combining these strategies remain unexplored. To further reduce
redundancy, we introduce HoliTom, a novel training-free holistic token merging
framework. HoliTom employs outer-LLM pruning through global redundancy-aware
temporal segmentation, followed by spatial-temporal merging to reduce visual
tokens by over 90%, significantly alleviating the LLM's computational burden.
Complementing this, we introduce a robust inner-LLM token similarity-based
merging approach, designed for superior performance and compatibility with
outer-LLM pruning. Evaluations demonstrate our method's promising
efficiency-performance trade-off on LLaVA-OneVision-7B, reducing computational
costs to 6.9% of FLOPs while maintaining 99.1% of the original performance.
Furthermore, we achieve a 2.28x reduction in Time-To-First-Token (TTFT) and a
1.32x acceleration in decoding throughput, highlighting the practical benefits
of our integrated pruning approach for efficient video LLMs inference.
HoliTom combines outer-LLM pruning through global temporal segmentation with inner-LLM token similarity-based merging to significantly reduce computational inefficiency in video LLMs without sacrificing performance.
video LLMs video tokens FastV inner-LLM pruning outer-LLM pruning global redundancy-aware temporal segmentation spatial-temporal merging HoliTom token similarity-based merging LLaVA-OneVision-7B Time-To-First-Token (TTFT) decoding throughput