Are We Using the Right Benchmark: An Evaluation Framework for Visual
  Token Compression Methods
Recent endeavors to accelerate inference in Multimodal Large Language Models
(MLLMs) have primarily focused on visual token compression. The effectiveness
of these methods is typically assessed by measuring the accuracy drop on
established benchmarks, comparing model performance before and after
compression. However, these benchmarks are originally designed to assess the
perception and reasoning capabilities of MLLMs, rather than to evaluate
compression techniques. As a result, directly applying them to visual token
compression introduces a task mismatch. Strikingly, our investigation reveals
that simple image downsampling consistently outperforms many advanced
compression methods across multiple widely used benchmarks. Through extensive
experiments, we make the following observations: (i) Current benchmarks are
noisy for the visual token compression task. (ii) Down-sampling is able to
serve as a data filter to evaluate the difficulty of samples in the visual
token compression task. Motivated by these findings, we introduce VTC-Bench, an
evaluation framework that incorporates a data filtering mechanism to denoise
existing benchmarks, thereby enabling fairer and more accurate assessment of
visual token compression methods. All data and code are available at
https://github.com/Chenfei-Liao/VTC-Bench.
68e750887ae125f9582e6a98
Chenfei Liao
claimed_verified
2025-10-17T04:34:40.932Z
6806464ed918f6d2fee2bc8b
https://cdn-avatars.huggingface.co/v1/production/uploads/6806464ed918f6d2fee2bc8b/rgpG2oO0m6PT0KltCF_Wf.jpeg
Chenfei Liao
user
Chenfei-Liao
68e750887ae125f9582e6a99
Wensong Wang
68e750887ae125f9582e6a9a
Zichen Wen
claimed_verified
2025-10-20T12:14:16.652Z
653b8c3e97a4d71d950e2f20
/avatars/b68880022e14556d0be58c69615db3be.svg
Zichen Wen
user
zichenwen
68e750887ae125f9582e6a9b
Xu Zheng
claimed_verified
2025-10-13T10:10:14.122Z
65961a7daa677adc97d8e33f
https://cdn-avatars.huggingface.co/v1/production/uploads/65961a7daa677adc97d8e33f/TA0Vz7CDmYZEdrhnsj_1b.jpeg
Xu Zheng
user
xz287
68e750887ae125f9582e6a9c
Yiyu Wang
68e750887ae125f9582e6a9d
Haocong He
68e750887ae125f9582e6a9e
Yuanhuiyi Lyu
68e750887ae125f9582e6a9f
Lutao Jiang
68e750887ae125f9582e6aa0
Xin Zou
68e750887ae125f9582e6aa1
Yuqian Fu
68e750887ae125f9582e6aa2
Bin Ren
68e750887ae125f9582e6aa3
Linfeng Zhang
68e750887ae125f9582e6aa4
Xuming Hu
Are We Using the Right Benchmark: An Evaluation Framework for Visual
  Token Compression Methods
Recent endeavors to accelerate inference in Multimodal Large Language Models
(MLLMs) have primarily focused on visual token compression. The effectiveness
of these methods is typically assessed by measuring the accuracy drop on
established benchmarks, comparing model performance before and after
compression. However, these benchmarks are originally designed to assess the
perception and reasoning capabilities of MLLMs, rather than to evaluate
compression techniques. As a result, directly applying them to visual token
compression introduces a task mismatch. Strikingly, our investigation reveals
that simple image downsampling consistently outperforms many advanced
compression methods across multiple widely used benchmarks. Through extensive
experiments, we make the following observations: (i) Current benchmarks are
noisy for the visual token compression task. (ii) Down-sampling is able to
serve as a data filter to evaluate the difficulty of samples in the visual
token compression task. Motivated by these findings, we introduce VTC-Bench, an
evaluation framework that incorporates a data filtering mechanism to denoise
existing benchmarks, thereby enabling fairer and more accurate assessment of
visual token compression methods. All data and code are available at
https://github.com/Chenfei-Liao/VTC-Bench.
VTC-Bench is introduced to provide a fair evaluation framework for visual token compression by incorporating a data filtering mechanism to denoise existing benchmarks.
Multimodal Large Language Models visual token compression image downsampling VTC-Bench data filtering mechanism