MMTok: Multimodal Coverage Maximization for Efficient Inference of VLMs
Vision-Language Models (VLMs) demonstrate impressive performance in
understanding visual content with language instruction by converting visual
input to vision tokens. However, redundancy in vision tokens results in the
degenerated inference efficiency of VLMs. While many algorithms have been
proposed to reduce the number of vision tokens, most of them apply only
unimodal information (i.e., vision/text) for pruning and ignore the inherent
multimodal property of vision-language tasks. Moreover, it lacks a generic
criterion that can be applied to different modalities. To mitigate this
limitation, in this work, we propose to leverage both vision and text tokens to
select informative vision tokens by the criterion of coverage. We first
formulate the subset selection problem as a maximum coverage problem.
Afterward, a subset of vision tokens is optimized to cover the text tokens and
the original set of vision tokens, simultaneously. Finally, a VLM agent can be
adopted to further improve the quality of text tokens for guiding vision
pruning. The proposed method MMTok is extensively evaluated on benchmark
datasets with different VLMs. The comparison illustrates that vision and text
information are complementary, and combining multimodal information can surpass
the unimodal baseline with a clear margin. Moreover, under the maximum coverage
criterion on the POPE dataset, our method achieves a 1.87x speedup while
maintaining 98.7% of the original performance on LLaVA-NeXT-13B. Furthermore,
with only four vision tokens, it still preserves 87.7% of the original
performance on LLaVA-1.5-7B. These results highlight the effectiveness of
coverage in token selection.
68ae0078364411bea07df667
Sixun Dong
claimed_verified
2025-08-27T07:11:12.675Z
653281ba1995cee54ac7ac4d
/avatars/077197d7b00502216b8edb1c87544791.svg
Sixun Dong
user
Ironieser
68ae0078364411bea07df668
Juhua Hu
68ae0078364411bea07df669
Mian Zhang
68ae0078364411bea07df66a
Ming Yin
claimed_verified
2025-08-27T07:11:14.599Z
66d4f80bbb76fb26ee6298cf
/avatars/b341445ecc7f9b1c63b4bcba94d83bd0.svg
Ming Yin
user
Kevin355
68ae0078364411bea07df66b
Yanjie Fu
68ae0078364411bea07df66c
Qi Qian
MMTok: Multimodal Coverage Maximization for Efficient Inference of VLMs
Vision-Language Models (VLMs) demonstrate impressive performance in
understanding visual content with language instruction by converting visual
input to vision tokens. However, redundancy in vision tokens results in the
degenerated inference efficiency of VLMs. While many algorithms have been
proposed to reduce the number of vision tokens, most of them apply only
unimodal information (i.e., vision/text) for pruning and ignore the inherent
multimodal property of vision-language tasks. Moreover, it lacks a generic
criterion that can be applied to different modalities. To mitigate this
limitation, in this work, we propose to leverage both vision and text tokens to
select informative vision tokens by the criterion of coverage. We first
formulate the subset selection problem as a maximum coverage problem.
Afterward, a subset of vision tokens is optimized to cover the text tokens and
the original set of vision tokens, simultaneously. Finally, a VLM agent can be
adopted to further improve the quality of text tokens for guiding vision
pruning. The proposed method MMTok is extensively evaluated on benchmark
datasets with different VLMs. The comparison illustrates that vision and text
information are complementary, and combining multimodal information can surpass
the unimodal baseline with a clear margin. Moreover, under the maximum coverage
criterion on the POPE dataset, our method achieves a 1.87x speedup while
maintaining 98.7% of the original performance on LLaVA-NeXT-13B. Furthermore,
with only four vision tokens, it still preserves 87.7% of the original
performance on LLaVA-1.5-7B. These results highlight the effectiveness of
coverage in token selection.
A multimodal method leverages both vision and text tokens to optimize vision token selection, improving inference efficiency in vision-language models.
vision tokens text tokens multimodal property maximum coverage problem VLM agent POPE dataset LLaVA-NeXT-13B LLaVA-1.5-7B