AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and
  Pruning
Large language models (LLMs) have enabled the creation of multi-modal LLMs
that exhibit strong comprehension of visual data such as images and videos.
However, these models usually rely on extensive visual tokens from visual
encoders, leading to high computational demands, which limits their
applicability in resource-constrained environments and for long-context tasks.
In this work, we propose a training-free adaptive inference method for
multi-modal LLMs that can accommodate a broad range of efficiency requirements
with a minimum performance drop. Our method consists of a) iterative token
merging based on embedding similarity before LLMs, and b) progressive token
pruning within LLM layers based on multi-modal importance. With a minimalist
design, our method can be applied to both video and image LLMs. Extensive
experiments on diverse video and image benchmarks demonstrate that, our method
substantially reduces computation load (e.g., a 7-fold reduction in
FLOPs) while preserving the performance of video and image LLMs. Further, under
a similar computational cost, our method outperforms the state-of-the-art
methods in long video understanding (e.g., +4.6 on MLVU).
Additionally, our in-depth analysis provides insights into token redundancy and
LLM layer behaviors, offering guidance for future research in designing
efficient multi-modal LLMs. Our code will be available at
https://github.com/LaVi-Lab/AIM.
67510af7a9d103be6ce725e0
Yiwu Zhong
67510af7a9d103be6ce725e1
Zhuoming Liu
claimed_verified
2025-03-19T09:48:28.448Z
666f2534a119281ee0e39783
/avatars/1de737da5cdf68d9bb1f98db8d5ad8b9.svg
Zhuoming Liu
user
zhuomingliu
67510af7a9d103be6ce725e2
Yin Li
67510af7a9d103be6ce725e3
Liwei Wang
AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and
  Pruning
Large language models (LLMs) have enabled the creation of multi-modal LLMs
that exhibit strong comprehension of visual data such as images and videos.
However, these models usually rely on extensive visual tokens from visual
encoders, leading to high computational demands, which limits their
applicability in resource-constrained environments and for long-context tasks.
In this work, we propose a training-free adaptive inference method for
multi-modal LLMs that can accommodate a broad range of efficiency requirements
with a minimum performance drop. Our method consists of a) iterative token
merging based on embedding similarity before LLMs, and b) progressive token
pruning within LLM layers based on multi-modal importance. With a minimalist
design, our method can be applied to both video and image LLMs. Extensive
experiments on diverse video and image benchmarks demonstrate that, our method
substantially reduces computation load (e.g., a 7-fold reduction in
FLOPs) while preserving the performance of video and image LLMs. Further, under
a similar computational cost, our method outperforms the state-of-the-art
methods in long video understanding (e.g., +4.6 on MLVU).
Additionally, our in-depth analysis provides insights into token redundancy and
LLM layer behaviors, offering guidance for future research in designing
efficient multi-modal LLMs. Our code will be available at
https://github.com/LaVi-Lab/AIM.
An adaptive inference method reduces computational demands in multi-modal LLMs for video and image understanding without significant performance loss.
large language models (LLMs) multi-modal LLMs iterative token merging embedding similarity progressive token pruning multi-modal importance FLOPs long video understanding token redundancy LLM layer behaviors