MIEB: Massive Image Embedding Benchmark
Image representations are often evaluated through disjointed, task-specific
protocols, leading to a fragmented understanding of model capabilities. For
instance, it is unclear whether an image embedding model adept at clustering
images is equally good at retrieving relevant images given a piece of text. We
introduce the Massive Image Embedding Benchmark (MIEB) to evaluate the
performance of image and image-text embedding models across the broadest
spectrum to date. MIEB spans 38 languages across 130 individual tasks, which we
group into 8 high-level categories. We benchmark 50 models across our
benchmark, finding that no single method dominates across all task categories.
We reveal hidden capabilities in advanced vision models such as their accurate
visual representation of texts, and their yet limited capabilities in
interleaved encodings and matching images and texts in the presence of
confounders. We also show that the performance of vision encoders on MIEB
correlates highly with their performance when used in multimodal large language
models. Our code, dataset, and leaderboard are publicly available at
https://github.com/embeddings-benchmark/mteb.
67fe0b5b45d0155d68b3c8c9
Chenghao Xiao
claimed_verified
2025-04-16T09:23:19.869Z
63108cc834c7d77420b0fd68
/avatars/2721e573a417a8ec0b81ee048c4b42ba.svg
chenghao xiao
user
gowitheflow
67fe0b5b45d0155d68b3c8ca
Isaac Chung
claimed_verified
2025-04-16T09:23:26.862Z
64cc0e80a257a3212c0c4b24
https://cdn-avatars.huggingface.co/v1/production/uploads/64cc0e80a257a3212c0c4b24/wqs6WZN8-3OQthcnQXgN7.png
Isaac Chung
user
isaacchung
67fe0b5b45d0155d68b3c8cb
Imene Kerboua
claimed_verified
2025-04-22T10:14:14.590Z
62610f8040e04009e81047e9
https://cdn-avatars.huggingface.co/v1/production/uploads/62610f8040e04009e81047e9/iqGTYB7OMmS0jkFEB7cEF.jpeg
Imene Kerboua
user
imenelydiaker
67fe0b5b45d0155d68b3c8cc
Jamie Stirling
67fe0b5b45d0155d68b3c8cd
Xin Zhang
claimed_verified
2025-04-15T07:52:33.782Z
63b6dbc8ccebeadccc888456
https://cdn-avatars.huggingface.co/v1/production/uploads/1673396893898-63b6dbc8ccebeadccc888456.jpeg
Xin Zhang
user
izhx
67fe0b5b45d0155d68b3c8ce
Márton Kardos
67fe0b5b45d0155d68b3c8cf
Roman Solomatin
claimed_verified
2025-04-16T09:23:24.757Z
61af4544d691b3aadd1f62b6
/avatars/7a4067accdd1005f78c3c4adad3ee0a5.svg
Solomatin Roman
user
Samoed
67fe0b5b45d0155d68b3c8d0
Noura Al Moubayed
67fe0b5b45d0155d68b3c8d1
Kenneth Enevoldsen
claimed_verified
2025-04-16T09:23:22.425Z
5ff5943752c26e9bc240bada
https://cdn-avatars.huggingface.co/v1/production/uploads/5ff5943752c26e9bc240bada/Exyzf3C_gJ2KdsL4K5_cq.png
Kenneth C. Enevoldsen
user
KennethEnevoldsen
67fe0b5b45d0155d68b3c8d2
Niklas Muennighoff
MIEB: Massive Image Embedding Benchmark
Image representations are often evaluated through disjointed, task-specific
protocols, leading to a fragmented understanding of model capabilities. For
instance, it is unclear whether an image embedding model adept at clustering
images is equally good at retrieving relevant images given a piece of text. We
introduce the Massive Image Embedding Benchmark (MIEB) to evaluate the
performance of image and image-text embedding models across the broadest
spectrum to date. MIEB spans 38 languages across 130 individual tasks, which we
group into 8 high-level categories. We benchmark 50 models across our
benchmark, finding that no single method dominates across all task categories.
We reveal hidden capabilities in advanced vision models such as their accurate
visual representation of texts, and their yet limited capabilities in
interleaved encodings and matching images and texts in the presence of
confounders. We also show that the performance of vision encoders on MIEB
correlates highly with their performance when used in multimodal large language
models. Our code, dataset, and leaderboard are publicly available at
https://github.com/embeddings-benchmark/mteb.
The Massive Image Embedding Benchmark (MIEB) evaluates image and image-text embedding models across various tasks, revealing hidden capabilities and performance correlations with multimodal large language models.
Massive Image Embedding Benchmark MIEB image embedding model text retrieval vision models image-text embedding multilingual task categories visual representation interleaved encodings multimodal large language models