I2CR: Intra- and Inter-modal Collaborative Reflections for Multimodal
  Entity Linking
Multimodal entity linking plays a crucial role in a wide range of
applications. Recent advances in large language model-based methods have become
the dominant paradigm for this task, effectively leveraging both textual and
visual modalities to enhance performance. Despite their success, these methods
still face two challenges, including unnecessary incorporation of image data in
certain scenarios and the reliance only on a one-time extraction of visual
features, which can undermine their effectiveness and accuracy. To address
these challenges, we propose a novel LLM-based framework for the multimodal
entity linking task, called Intra- and Inter-modal Collaborative Reflections.
This framework prioritizes leveraging text information to address the task.
When text alone is insufficient to link the correct entity through intra- and
inter-modality evaluations, it employs a multi-round iterative strategy that
integrates key visual clues from various aspects of the image to support
reasoning and enhance matching accuracy. Extensive experiments on three widely
used public datasets demonstrate that our framework consistently outperforms
current state-of-the-art methods in the task, achieving improvements of 3.2%,
5.1%, and 1.6%, respectively. Our code is available at
https://github.com/ziyan-xiaoyu/I2CR/.
6895ca3fe78b624607aa9672
Ziyan Liu
6895ca3fe78b624607aa9673
Junwen Li
6895ca3fe78b624607aa9674
Kaiwen Li
6895ca3fe78b624607aa9675
Tong Ruan
6895ca3fe78b624607aa9676
Chao Wang
6895ca3fe78b624607aa9677
Xinyan He
6895ca3fe78b624607aa9678
Zongyu Wang
6895ca3fe78b624607aa9679
Xuezhi Cao
6895ca3fe78b624607aa967a
Jingping Liu
I2CR: Intra- and Inter-modal Collaborative Reflections for Multimodal
  Entity Linking
Multimodal entity linking plays a crucial role in a wide range of
applications. Recent advances in large language model-based methods have become
the dominant paradigm for this task, effectively leveraging both textual and
visual modalities to enhance performance. Despite their success, these methods
still face two challenges, including unnecessary incorporation of image data in
certain scenarios and the reliance only on a one-time extraction of visual
features, which can undermine their effectiveness and accuracy. To address
these challenges, we propose a novel LLM-based framework for the multimodal
entity linking task, called Intra- and Inter-modal Collaborative Reflections.
This framework prioritizes leveraging text information to address the task.
When text alone is insufficient to link the correct entity through intra- and
inter-modality evaluations, it employs a multi-round iterative strategy that
integrates key visual clues from various aspects of the image to support
reasoning and enhance matching accuracy. Extensive experiments on three widely
used public datasets demonstrate that our framework consistently outperforms
current state-of-the-art methods in the task, achieving improvements of 3.2%,
5.1%, and 1.6%, respectively. Our code is available at
https://github.com/ziyan-xiaoyu/I2CR/.
A novel LLM-based framework, Intra- and Inter-modal Collaborative Reflections, enhances multimodal entity linking by prioritizing text and using iterative visual clues when necessary, outperforming current state-of-the-art methods.
LLM-based methods multimodal entity linking intra-modal inter-modal collaborative reflections iterative strategy visual clues reasoning matching accuracy