Multimodal Referring Segmentation: A Survey
Multimodal referring segmentation aims to segment target objects in visual
scenes, such as images, videos, and 3D scenes, based on referring expressions
in text or audio format. This task plays a crucial role in practical
applications requiring accurate object perception based on user instructions.
Over the past decade, it has gained significant attention in the multimodal
community, driven by advances in convolutional neural networks, transformers,
and large language models, all of which have substantially improved multimodal
perception capabilities. This paper provides a comprehensive survey of
multimodal referring segmentation. We begin by introducing this field's
background, including problem definitions and commonly used datasets. Next, we
summarize a unified meta architecture for referring segmentation and review
representative methods across three primary visual scenes, including images,
videos, and 3D scenes. We further discuss Generalized Referring Expression
(GREx) methods to address the challenges of real-world complexity, along with
related tasks and practical applications. Extensive performance comparisons on
standard benchmarks are also provided. We continually track related works at
https://github.com/henghuiding/Awesome-Multimodal-Referring-Segmentation.
6890159b0a411b3b8d28d650
Henghui Ding
6890159b0a411b3b8d28d651
Song Tang
6890159b0a411b3b8d28d652
Shuting He
6890159b0a411b3b8d28d653
Chang Liu
6890159b0a411b3b8d28d654
Zuxuan Wu
6890159b0a411b3b8d28d655
Yu-Gang Jiang
Multimodal Referring Segmentation: A Survey
Multimodal referring segmentation aims to segment target objects in visual
scenes, such as images, videos, and 3D scenes, based on referring expressions
in text or audio format. This task plays a crucial role in practical
applications requiring accurate object perception based on user instructions.
Over the past decade, it has gained significant attention in the multimodal
community, driven by advances in convolutional neural networks, transformers,
and large language models, all of which have substantially improved multimodal
perception capabilities. This paper provides a comprehensive survey of
multimodal referring segmentation. We begin by introducing this field's
background, including problem definitions and commonly used datasets. Next, we
summarize a unified meta architecture for referring segmentation and review
representative methods across three primary visual scenes, including images,
videos, and 3D scenes. We further discuss Generalized Referring Expression
(GREx) methods to address the challenges of real-world complexity, along with
related tasks and practical applications. Extensive performance comparisons on
standard benchmarks are also provided. We continually track related works at
https://github.com/henghuiding/Awesome-Multimodal-Referring-Segmentation.
A survey of multimodal referring segmentation techniques, covering advancements in convolutional neural networks, transformers, and large language models for segmenting objects in images, videos, and 3D scenes based on text or audio instructions.
convolutional neural networks transformers large language models multimodal referring segmentation Generalized Referring Expression (GREx)