VLM2Vec: Training Vision-Language Models for Massive Multimodal
  Embedding Tasks
Embedding models have been crucial in enabling various downstream tasks such
as semantic similarity, information retrieval, and clustering. Recently, there
has been a surge of interest in developing universal text embedding models that
can generalize across tasks (e.g., MTEB). However, progress in learning
universal multimodal embedding models has been relatively slow despite their
importance. In this work, we aim to explore the potential for building
universal embeddings capable of handling a wide range of downstream tasks. Our
contributions are twofold: (1) MMEB (Massive Multimodal Embedding Benchmark),
which covers 4 meta-tasks (i.e. classification, visual question answering,
multimodal retrieval, and visual grounding) and 36 datasets, including 20
training and 16 evaluation datasets, and (2) VLM2Vec (Vision-Language Model ->
Vector), a contrastive training framework that converts any state-of-the-art
vision-language model into an embedding model via training on MMEB. Unlike
previous models such as CLIP and BLIP, VLM2Vec can process any combination of
images and text to generate a fixed-dimensional vector based on task
instructions. We build a series of VLM2Vec models on Phi-3.5-V and evaluate
them on MMEB's evaluation split. Our results show that \model achieves an
absolute average improvement of 10% to 20% over existing multimodal embedding
models on both in-distribution and out-of-distribution datasets in MMEB.
6708056d0890f205c2aa0353
Ziyan Jiang
extracted_confirmed
2025-07-08T07:54:40.227Z
64778fb8168cb428e00f69b0
https://cdn-avatars.huggingface.co/v1/production/uploads/64778fb8168cb428e00f69b0/D_XYg74zHG9K3HUJj_gD4.jpeg
Ziyan Jiang
user
ziyjiang
6708056d0890f205c2aa0354
Rui Meng
6708056d0890f205c2aa0355
Xinyi Yang
6708056d0890f205c2aa0356
Semih Yavuz
6708056d0890f205c2aa0357
Yingbo Zhou
6708056d0890f205c2aa0358
Wenhu Chen
extracted_pending
2024-10-10T16:48:47.379Z
6313a86154e6e5d9f0f94e04
https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
Wenhu Chen
user
wenhu
VLM2Vec: Training Vision-Language Models for Massive Multimodal
  Embedding Tasks
Embedding models have been crucial in enabling various downstream tasks such
as semantic similarity, information retrieval, and clustering. Recently, there
has been a surge of interest in developing universal text embedding models that
can generalize across tasks (e.g., MTEB). However, progress in learning
universal multimodal embedding models has been relatively slow despite their
importance. In this work, we aim to explore the potential for building
universal embeddings capable of handling a wide range of downstream tasks. Our
contributions are twofold: (1) MMEB (Massive Multimodal Embedding Benchmark),
which covers 4 meta-tasks (i.e. classification, visual question answering,
multimodal retrieval, and visual grounding) and 36 datasets, including 20
training and 16 evaluation datasets, and (2) VLM2Vec (Vision-Language Model ->
Vector), a contrastive training framework that converts any state-of-the-art
vision-language model into an embedding model via training on MMEB. Unlike
previous models such as CLIP and BLIP, VLM2Vec can process any combination of
images and text to generate a fixed-dimensional vector based on task
instructions. We build a series of VLM2Vec models on Phi-3.5-V and evaluate
them on MMEB's evaluation split. Our results show that \model achieves an
absolute average improvement of 10% to 20% over existing multimodal embedding
models on both in-distribution and out-of-distribution datasets in MMEB.
A new framework, VLM2Vec, converts vision-language models into universal multimodal embedding models using a large benchmark, MMEB, achieving significant improvements across various tasks.
embedding models semantic similarity information retrieval clustering universal text embedding models MTEB universal multimodal embedding models MMEB (Massive Multimodal Embedding Benchmark) VLM2Vec contrastive training framework vision-language model fixed-dimensional vector in-distribution out-of-distribution datasets