Introducing Visual Perception Token into Multimodal Large Language Model
To utilize visual information, Multimodal Large Language Model (MLLM) relies
on the perception process of its vision encoder. The completeness and accuracy
of visual perception significantly influence the precision of spatial
reasoning, fine-grained understanding, and other tasks. However, MLLM still
lacks the autonomous capability to control its own visual perception processes,
for example, selectively reviewing specific regions of an image or focusing on
information related to specific object categories. In this work, we propose the
concept of Visual Perception Token, aiming to empower MLLM with a mechanism to
control its visual perception processes. We design two types of Visual
Perception Tokens, termed the Region Selection Token and the Vision Re-Encoding
Token. MLLMs autonomously generate these tokens, just as they generate text,
and use them to trigger additional visual perception actions. The Region
Selection Token explicitly identifies specific regions in an image that require
further perception, while the Vision Re-Encoding Token uses its hidden states
as control signals to guide additional visual perception processes. Extensive
experiments demonstrate the advantages of these tokens in handling spatial
reasoning, improving fine-grained understanding, and other tasks. On average,
the introduction of Visual Perception Tokens improves the performance of a 2B
model by 23.6\%, increasing its score from 0.572 to 0.708, and even outperforms
a 7B parameter model by 13.4\% (from 0.624). Please check out our repo
https://github.com/yu-rp/VisualPerceptionToken
67bddd63c7d8b835b82ced9a
Runpeng Yu
admin_assigned
2025-02-26T09:14:07.580Z
635364b3c41f548fe39db945
/avatars/ad1916bbfabca0b6651c8eabacc5eba8.svg
Runpeng Yu
user
rp-yu
67bddd63c7d8b835b82ced9b
Xinyin Ma
admin_assigned
2025-02-26T09:14:13.670Z
64396ebc21221ac7411852b3
https://cdn-avatars.huggingface.co/v1/production/uploads/64396ebc21221ac7411852b3/SR0dC8N0bdj9tZFxYPpSf.jpeg
Xinyin Ma
user
horseee
67bddd63c7d8b835b82ced9c
Xinchao Wang
admin_assigned
2025-02-26T09:14:53.838Z
63fc03a50aab060792ffef39
/avatars/9d5b1bb2a41928e08176b703935133ab.svg
Wangxinchao
user
wxcTest
Introducing Visual Perception Token into Multimodal Large Language Model
To utilize visual information, Multimodal Large Language Model (MLLM) relies
on the perception process of its vision encoder. The completeness and accuracy
of visual perception significantly influence the precision of spatial
reasoning, fine-grained understanding, and other tasks. However, MLLM still
lacks the autonomous capability to control its own visual perception processes,
for example, selectively reviewing specific regions of an image or focusing on
information related to specific object categories. In this work, we propose the
concept of Visual Perception Token, aiming to empower MLLM with a mechanism to
control its visual perception processes. We design two types of Visual
Perception Tokens, termed the Region Selection Token and the Vision Re-Encoding
Token. MLLMs autonomously generate these tokens, just as they generate text,
and use them to trigger additional visual perception actions. The Region
Selection Token explicitly identifies specific regions in an image that require
further perception, while the Vision Re-Encoding Token uses its hidden states
as control signals to guide additional visual perception processes. Extensive
experiments demonstrate the advantages of these tokens in handling spatial
reasoning, improving fine-grained understanding, and other tasks. On average,
the introduction of Visual Perception Tokens improves the performance of a 2B
model by 23.6\%, increasing its score from 0.572 to 0.708, and even outperforms
a 7B parameter model by 13.4\% (from 0.624). Please check out our repo
https://github.com/yu-rp/VisualPerceptionToken
Visual Perception Tokens improve Multimodal Large Language Models' control over visual perception processes, enhancing tasks like spatial reasoning and fine-grained understanding.
Multimodal Large Language Model vision encoder Visual Perception Token Region Selection Token Vision Re-Encoding Token