VIRTUE: Visual-Interactive Text-Image Universal Embedder
Multimodal representation learning models have demonstrated successful
operation across complex tasks, and the integration of vision-language models
(VLMs) has further enabled embedding models with instruction-following
capabilities. However, existing embedding models lack visual-interactive
capabilities to specify regions of interest from users (e.g., point, bounding
box, mask), which have been explored in generative models to broaden their
human-interactive applicability. Equipping embedding models with visual
interactions not only would unlock new applications with localized grounding of
user intent, which remains unexplored, but also enable the models to learn
entity-level information within images to complement their global
representations for conventional embedding tasks. In this paper, we propose a
novel Visual-InteRactive Text-Image Universal Embedder (VIRTUE) that extends
the capabilities of the segmentation model and the vision-language model to the
realm of representation learning. In VIRTUE, the segmentation model can process
visual prompts that pinpoint specific regions within an image, thereby enabling
the embedder to handle complex and ambiguous scenarios more precisely. To
evaluate the visual-interaction ability of VIRTUE, we introduce a large-scale
Segmentation-and-Scene Caption Retrieval (SCaR) benchmark comprising 1M samples
that aims to retrieve the text caption by jointly considering the entity with a
specific object and image scene. VIRTUE consistently achieves a
state-of-the-art performance with significant improvements across 36 universal
MMEB (3.1%-8.5%) and five visual-interactive SCaR (15.2%-20.3%) tasks.
68df2bccdf49fb0df1e03bed
Wei-Yao Wang
68df2bccdf49fb0df1e03bee
Kazuya Tateishi
68df2bccdf49fb0df1e03bef
Qiyu Wu
68df2bccdf49fb0df1e03bf0
Shusuke Takahashi
68df2bccdf49fb0df1e03bf1
Yuki Mitsufuji
VIRTUE: Visual-Interactive Text-Image Universal Embedder
Multimodal representation learning models have demonstrated successful
operation across complex tasks, and the integration of vision-language models
(VLMs) has further enabled embedding models with instruction-following
capabilities. However, existing embedding models lack visual-interactive
capabilities to specify regions of interest from users (e.g., point, bounding
box, mask), which have been explored in generative models to broaden their
human-interactive applicability. Equipping embedding models with visual
interactions not only would unlock new applications with localized grounding of
user intent, which remains unexplored, but also enable the models to learn
entity-level information within images to complement their global
representations for conventional embedding tasks. In this paper, we propose a
novel Visual-InteRactive Text-Image Universal Embedder (VIRTUE) that extends
the capabilities of the segmentation model and the vision-language model to the
realm of representation learning. In VIRTUE, the segmentation model can process
visual prompts that pinpoint specific regions within an image, thereby enabling
the embedder to handle complex and ambiguous scenarios more precisely. To
evaluate the visual-interaction ability of VIRTUE, we introduce a large-scale
Segmentation-and-Scene Caption Retrieval (SCaR) benchmark comprising 1M samples
that aims to retrieve the text caption by jointly considering the entity with a
specific object and image scene. VIRTUE consistently achieves a
state-of-the-art performance with significant improvements across 36 universal
MMEB (3.1%-8.5%) and five visual-interactive SCaR (15.2%-20.3%) tasks.
VIRTUE, a novel Visual-InteRactive Text-Image Universal Embedder, integrates segmentation and vision-language models to enable visual interactions and localized grounding, achieving state-of-the-art performance in representation learning tasks.
multimodal representation learning vision-language models visual-interactive capabilities segmentation model visual prompts entity-level information global representations Segmentation-and-Scene Caption Retrieval MMEB SCaR