UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learning
Universal multimodal embedding models are foundational to various tasks.
Existing approaches typically employ in-batch negative mining by measuring the
similarity of query-candidate pairs. However, these methods often struggle to
capture subtle semantic differences among candidates and lack diversity in
negative samples. Moreover, the embeddings exhibit limited discriminative
ability in distinguishing false and hard negatives. In this paper, we leverage
the advanced understanding capabilities of MLLMs to enhance representation
learning and present a novel Universal Multimodal Embedding (UniME-V2) model.
Our approach first constructs a potential hard negative set through global
retrieval. We then introduce the MLLM-as-a-Judge mechanism, which utilizes
MLLMs to assess the semantic alignment of query-candidate pairs and generate
soft semantic matching scores. These scores serve as a foundation for hard
negative mining, mitigating the impact of false negatives and enabling the
identification of diverse, high-quality hard negatives. Furthermore, the
semantic matching scores are used as soft labels to mitigate the rigid
one-to-one mapping constraint. By aligning the similarity matrix with the soft
semantic matching score matrix, the model learns semantic distinctions among
candidates, significantly enhancing its discriminative capacity. To further
improve performance, we propose UniME-V2-Reranker, a reranking model trained on
our mined hard negatives through a joint pairwise and listwise optimization
approach. We conduct comprehensive experiments on the MMEB benchmark and
multiple retrieval tasks, demonstrating that our method achieves
state-of-the-art performance on average across all tasks.
68f052c836f8b025381e190a
Tiancheng Gu
claimed_verified
2025-10-16T11:40:30.471Z
641030c77a15af878ae5bd8f
/avatars/8a5037edf55c78ebc317c8b191343671.svg
TianchengGu
user
TianchengGu
68f052c836f8b025381e190b
Kaicheng Yang
claimed_verified
2025-10-16T06:39:01.782Z
63e202f352b7578dba448ab5
https://cdn-avatars.huggingface.co/v1/production/uploads/63e202f352b7578dba448ab5/8itVBLcv14m7OVsoF8h1o.jpeg
Yang
user
Kaichengalex
68f052c836f8b025381e190c
Kaichen Zhang
68f052c836f8b025381e190d
Xiang An
claimed_verified
2025-10-21T14:06:17.757Z
6478679d7b370854241b2ad8
https://cdn-avatars.huggingface.co/v1/production/uploads/6478679d7b370854241b2ad8/dBczWYYdfEt9tQcnVGhQk.jpeg
xiangan
user
xiangan
68f052c836f8b025381e190e
Ziyong Feng
68f052c836f8b025381e190f
Yueyi Zhang
68f052c836f8b025381e1910
Weidong Cai
68f052c836f8b025381e1911
Jiankang Deng
68f052c836f8b025381e1912
Lidong Bing
UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learning
Universal multimodal embedding models are foundational to various tasks.
Existing approaches typically employ in-batch negative mining by measuring the
similarity of query-candidate pairs. However, these methods often struggle to
capture subtle semantic differences among candidates and lack diversity in
negative samples. Moreover, the embeddings exhibit limited discriminative
ability in distinguishing false and hard negatives. In this paper, we leverage
the advanced understanding capabilities of MLLMs to enhance representation
learning and present a novel Universal Multimodal Embedding (UniME-V2) model.
Our approach first constructs a potential hard negative set through global
retrieval. We then introduce the MLLM-as-a-Judge mechanism, which utilizes
MLLMs to assess the semantic alignment of query-candidate pairs and generate
soft semantic matching scores. These scores serve as a foundation for hard
negative mining, mitigating the impact of false negatives and enabling the
identification of diverse, high-quality hard negatives. Furthermore, the
semantic matching scores are used as soft labels to mitigate the rigid
one-to-one mapping constraint. By aligning the similarity matrix with the soft
semantic matching score matrix, the model learns semantic distinctions among
candidates, significantly enhancing its discriminative capacity. To further
improve performance, we propose UniME-V2-Reranker, a reranking model trained on
our mined hard negatives through a joint pairwise and listwise optimization
approach. We conduct comprehensive experiments on the MMEB benchmark and
multiple retrieval tasks, demonstrating that our method achieves
state-of-the-art performance on average across all tasks.
A novel Universal Multimodal Embedding (UniME-V2) model uses MLLMs to enhance representation learning by identifying diverse, high-quality hard negatives and improving discriminative capacity through soft semantic matching scores.
Universal Multimodal Embedding UniME-V2 MLLMs in-batch negative mining global retrieval hard negative mining semantic alignment soft semantic matching scores soft labels joint pairwise and listwise optimization MMEB benchmark retrieval tasks