ABC: Achieving Better Control of Multimodal Embeddings using VLMs
Visual embedding models excel at zero-shot tasks like visual retrieval and
classification. However, these models cannot be used for tasks that contain
ambiguity or require user instruction. These tasks necessitate a multimodal
embedding model, which outputs embeddings that combine visual and natural
language input. Existing CLIP-based approaches embed images and text
independently, and fuse the result. We find that this results in weak
interactions between modalities, and poor user control over the representation.
We introduce ABC, an open-source multimodal embedding model that uses a
vision-language model backbone to deeply integrate image features with natural
language instructions. ABC achieves bestfor-size performance on MSCOCO
image-to-text retrieval and is the top performing model on classification and
VQA tasks in the Massive Multimodal Embedding Benchmark. With a strongly
unified vision-language representation, ABC can use natural language to solve
subtle and potentially ambiguous visual retrieval problems. To evaluate this
capability, we design CtrlBench, a benchmark that requires interleaving textual
instructions with image content for correct retrieval. ABC advances the state
of multimodal embeddings by offering high-quality representations and flexible
natural language control. Our model and datasets are available at our project
page.
67c755f898a2e37274c62c96
Benjamin Schneider
claimed_verified
2025-03-06T19:50:52.715Z
6669ffeecf27eed0f2f2581e
/avatars/6a1012cce15da12fa3bd6f9e0f87d100.svg
Benjamin Schneider
user
BenSchneider
67c755f898a2e37274c62c97
Florian Kerschbaum
67c755f898a2e37274c62c98
Wenhu Chen
admin_assigned
2025-03-06T09:50:07.881Z
6313a86154e6e5d9f0f94e04
https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
Wenhu Chen
user
wenhu
ABC: Achieving Better Control of Multimodal Embeddings using VLMs
Visual embedding models excel at zero-shot tasks like visual retrieval and
classification. However, these models cannot be used for tasks that contain
ambiguity or require user instruction. These tasks necessitate a multimodal
embedding model, which outputs embeddings that combine visual and natural
language input. Existing CLIP-based approaches embed images and text
independently, and fuse the result. We find that this results in weak
interactions between modalities, and poor user control over the representation.
We introduce ABC, an open-source multimodal embedding model that uses a
vision-language model backbone to deeply integrate image features with natural
language instructions. ABC achieves bestfor-size performance on MSCOCO
image-to-text retrieval and is the top performing model on classification and
VQA tasks in the Massive Multimodal Embedding Benchmark. With a strongly
unified vision-language representation, ABC can use natural language to solve
subtle and potentially ambiguous visual retrieval problems. To evaluate this
capability, we design CtrlBench, a benchmark that requires interleaving textual
instructions with image content for correct retrieval. ABC advances the state
of multimodal embeddings by offering high-quality representations and flexible
natural language control. Our model and datasets are available at our project
page.
ABC is a multimodal embedding model using a vision-language backbone that achieves top performance on MSCOCO, classification, and VQA tasks, offering high-quality representations with flexible natural language control.
visual embedding models zero-shot tasks multimodal embedding model CLIP-based approaches image features natural language instructions MSCOCO image-to-text retrieval classification VQA tasks Massive Multimodal Embedding Benchmark visual retrieval problems CtrlBench