Visual Representation Alignment for Multimodal Large Language Models
Multimodal large language models (MLLMs) trained with visual instruction
tuning have achieved strong performance across diverse tasks, yet they remain
limited in vision-centric tasks such as object counting or spatial reasoning.
We attribute this gap to the prevailing text-only supervision paradigm, which
provides only indirect guidance for the visual pathway and often leads MLLMs to
discard fine-grained visual details during training. In this paper, we present
VIsual Representation ALignment (VIRAL), a simple yet effective regularization
strategy that aligns the internal visual representations of MLLMs with those of
pre-trained vision foundation models (VFMs). By explicitly enforcing this
alignment, VIRAL enables the model not only to retain critical visual details
from the input vision encoder but also to complement additional visual
knowledge from VFMs, thereby enhancing its ability to reason over complex
visual inputs. Our experiments demonstrate consistent improvements across all
tasks on widely adopted multimodal benchmarks. Furthermore, we conduct
comprehensive ablation studies to validate the key design choices underlying
our framework. We believe this simple finding opens up an important direction
for the effective integration of visual information in training MLLMs.
68c0d8f13912ed54cf543216
Heeji Yoon
68c0d8f13912ed54cf543217
Jaewoo Jung
claimed_verified
2025-09-12T16:18:21.076Z
65d02dc017e2b305e0d7bf4f
/avatars/2a50fd0541e7b0e200c577a661956696.svg
Jaewoo Jung
user
crepejung00
68c0d8f13912ed54cf543218
Junwan Kim
claimed_verified
2025-09-12T16:18:18.280Z
64c2c45ae818eec6128fdda3
/avatars/d4399e25e6399345e263c7902789047e.svg
Junwan Kim
user
junwann
68c0d8f13912ed54cf543219
Hyungyu Choi
claimed_verified
2025-09-15T15:09:07.426Z
68664cc383532f3fd1addf9e
https://cdn-avatars.huggingface.co/v1/production/uploads/68664cc383532f3fd1addf9e/W8vPyx1XtYXLIV1Y9JRCh.jpeg
Hyungyu Choi
user
qkenr0804
68c0d8f13912ed54cf54321a
Heeseong Shin
68c0d8f13912ed54cf54321b
Sangbeom Lim
68c0d8f13912ed54cf54321c
Honggyu An
68c0d8f13912ed54cf54321d
Chaehyun Kim
68c0d8f13912ed54cf54321e
Jisang Han
68c0d8f13912ed54cf54321f
Donghyun Kim
68c0d8f13912ed54cf543220
Chanho Eom
68c0d8f13912ed54cf543221
Sunghwan Hong
68c0d8f13912ed54cf543222
Seungryong Kim
claimed_verified
2025-09-12T16:18:15.000Z
6752b5ebebb87145beedaecb
/avatars/1de059e88dad6fe070cb22ba96d32914.svg
Seungryong Kim
user
seungryongkim
Visual Representation Alignment for Multimodal Large Language Models
Multimodal large language models (MLLMs) trained with visual instruction
tuning have achieved strong performance across diverse tasks, yet they remain
limited in vision-centric tasks such as object counting or spatial reasoning.
We attribute this gap to the prevailing text-only supervision paradigm, which
provides only indirect guidance for the visual pathway and often leads MLLMs to
discard fine-grained visual details during training. In this paper, we present
VIsual Representation ALignment (VIRAL), a simple yet effective regularization
strategy that aligns the internal visual representations of MLLMs with those of
pre-trained vision foundation models (VFMs). By explicitly enforcing this
alignment, VIRAL enables the model not only to retain critical visual details
from the input vision encoder but also to complement additional visual
knowledge from VFMs, thereby enhancing its ability to reason over complex
visual inputs. Our experiments demonstrate consistent improvements across all
tasks on widely adopted multimodal benchmarks. Furthermore, we conduct
comprehensive ablation studies to validate the key design choices underlying
our framework. We believe this simple finding opens up an important direction
for the effective integration of visual information in training MLLMs.
VIRAL, a regularization strategy, aligns MLLMs' visual representations with pre-trained VFMs, enhancing performance on vision-centric tasks.
multimodal large language models MLLMs visual instruction tuning vision-centric tasks object counting spatial reasoning text-only supervision visual pathway fine-grained visual details Visual Representation ALignment VIRAL pre-trained vision foundation models VFMs internal visual representations visual encoder visual knowledge reasoning over complex visual inputs multimodal benchmarks ablation studies