True Multimodal In-Context Learning Needs Attention to the Visual
  Context
Multimodal Large Language Models (MLLMs), built on powerful language
backbones, have enabled Multimodal In-Context Learning (MICL)-adapting to new
tasks from a few multimodal demonstrations consisting of images, questions, and
answers. Despite showing noticeable improvement on standard vision-language
datasets, current MLLMs struggle to leverage visual information in the
demonstrations. Specifically, they tend to neglect visual cues and over-rely on
textual patterns, leading to mere text imitation rather than genuine multimodal
adaptation. This behavior makes MICL still unimodal and largely restricts its
practical utility. More importantly, this limitation is often concealed by the
improved performance on tasks that do not require understanding the visual
context. As a result, how to effectively enhance MICL ability and reliably
evaluate the MICL performance remains underexplored. To address these issues,
we first introduce Dynamic Attention Reallocation (DARA), an efficient
fine-tuning strategy that encourages models to attend to the visual context by
rebalancing attention across visual and textual tokens. In addition, we present
TrueMICL, an MICL-dedicated dataset with both support and test sets that
explicitly requires the integration of multimodal information-particularly
visual content-for correct task completion. Extensive experiments demonstrate
the effectiveness of our holistic solution, showcasing substantial improvements
in the true multimodal in-context learning capabilities. Code and datasets are
available at https://chenxshuo.github.io/true-micl-colm .
6883fc897d7a19a208cdef55
Shuo Chen
6883fc897d7a19a208cdef56
Jianzhe Liu
6883fc897d7a19a208cdef57
Zhen Han
6883fc897d7a19a208cdef58
Yan Xia
6883fc897d7a19a208cdef59
Daniel Cremers
6883fc897d7a19a208cdef5a
Philip Torr
6883fc897d7a19a208cdef5b
Volker Tresp
6883fc897d7a19a208cdef5c
Jindong Gu
True Multimodal In-Context Learning Needs Attention to the Visual
  Context
Multimodal Large Language Models (MLLMs), built on powerful language
backbones, have enabled Multimodal In-Context Learning (MICL)-adapting to new
tasks from a few multimodal demonstrations consisting of images, questions, and
answers. Despite showing noticeable improvement on standard vision-language
datasets, current MLLMs struggle to leverage visual information in the
demonstrations. Specifically, they tend to neglect visual cues and over-rely on
textual patterns, leading to mere text imitation rather than genuine multimodal
adaptation. This behavior makes MICL still unimodal and largely restricts its
practical utility. More importantly, this limitation is often concealed by the
improved performance on tasks that do not require understanding the visual
context. As a result, how to effectively enhance MICL ability and reliably
evaluate the MICL performance remains underexplored. To address these issues,
we first introduce Dynamic Attention Reallocation (DARA), an efficient
fine-tuning strategy that encourages models to attend to the visual context by
rebalancing attention across visual and textual tokens. In addition, we present
TrueMICL, an MICL-dedicated dataset with both support and test sets that
explicitly requires the integration of multimodal information-particularly
visual content-for correct task completion. Extensive experiments demonstrate
the effectiveness of our holistic solution, showcasing substantial improvements
in the true multimodal in-context learning capabilities. Code and datasets are
available at https://chenxshuo.github.io/true-micl-colm .
Dynamic Attention Reallocation (DARA) and TrueMICL improve multimodal in-context learning by enhancing visual context integration and providing a dedicated evaluation dataset.
Multimodal Large Language Models Multimodal In-Context Learning Dynamic Attention Reallocation TrueMICL multimodal information visual context in-context learning capabilities