ViCO: A Training Strategy towards Semantic Aware Dynamic High-Resolution
Existing Multimodal Large Language Models (MLLMs) suffer from increased
inference costs due to the additional vision tokens introduced by image inputs.
In this work, we propose Visual Consistency Learning (ViCO), a novel training
algorithm that enables the model to represent images of varying semantic
complexities using different numbers of vision tokens. The key idea behind our
method is to employ multiple MLP connectors, each with a different image
compression ratio, to downsample the vision tokens based on the semantic
complexity of the image. During training, we minimize the KL divergence between
the responses conditioned on different MLP connectors. At inference time, we
introduce an image router, termed Visual Resolution Router (ViR), that
automatically selects the appropriate compression rate for each image patch.
Compared with existing dynamic high-resolution strategies, which adjust the
number of visual tokens based on image resolutions, our method dynamically
adapts the number of visual tokens according to semantic complexity.
Experimental results demonstrate that our method can reduce the number of
vision tokens by up to 50% while maintaining the model's perception, reasoning,
and OCR capabilities. We hope this work will contribute to the development of
more efficient MLLMs. The code and models will be released to facilitate future
research.
68ef2552486b78128f0e34f0
Long Cui
68ef2552486b78128f0e34f1
Weiyun Wang
68ef2552486b78128f0e34f2
Jie Shao
68ef2552486b78128f0e34f3
Zichen Wen
claimed_verified
2025-10-20T12:13:45.468Z
653b8c3e97a4d71d950e2f20
/avatars/b68880022e14556d0be58c69615db3be.svg
Zichen Wen
user
zichenwen
68ef2552486b78128f0e34f4
Gen Luo
68ef2552486b78128f0e34f5
Linfeng Zhang
68ef2552486b78128f0e34f6
Yanting Zhang
68ef2552486b78128f0e34f7
Yu Qiao
68ef2552486b78128f0e34f8
Wenhai Wang
ViCO: A Training Strategy towards Semantic Aware Dynamic High-Resolution
Existing Multimodal Large Language Models (MLLMs) suffer from increased
inference costs due to the additional vision tokens introduced by image inputs.
In this work, we propose Visual Consistency Learning (ViCO), a novel training
algorithm that enables the model to represent images of varying semantic
complexities using different numbers of vision tokens. The key idea behind our
method is to employ multiple MLP connectors, each with a different image
compression ratio, to downsample the vision tokens based on the semantic
complexity of the image. During training, we minimize the KL divergence between
the responses conditioned on different MLP connectors. At inference time, we
introduce an image router, termed Visual Resolution Router (ViR), that
automatically selects the appropriate compression rate for each image patch.
Compared with existing dynamic high-resolution strategies, which adjust the
number of visual tokens based on image resolutions, our method dynamically
adapts the number of visual tokens according to semantic complexity.
Experimental results demonstrate that our method can reduce the number of
vision tokens by up to 50% while maintaining the model's perception, reasoning,
and OCR capabilities. We hope this work will contribute to the development of
more efficient MLLMs. The code and models will be released to facilitate future
research.