MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional
  Multimodal Embeddings
Multimodal embedding models, built upon causal Vision Language Models (VLMs),
have shown promise in various tasks. However, current approaches face three key
limitations: the use of causal attention in VLM backbones is suboptimal for
embedding tasks; scalability issues due to reliance on high-quality labeled
paired data for contrastive learning; and limited diversity in training
objectives and data. To address these issues, we propose MoCa, a two-stage
framework for transforming pre-trained VLMs into effective bidirectional
multimodal embedding models. The first stage, Modality-aware Continual
Pre-training, introduces a joint reconstruction objective that simultaneously
denoises interleaved text and image inputs, enhancing bidirectional
context-aware reasoning. The second stage, Heterogeneous Contrastive
Fine-tuning, leverages diverse, semantically rich multimodal data beyond simple
image-caption pairs to enhance generalization and alignment. Our method
addresses the stated limitations by introducing bidirectional attention through
continual pre-training, scaling effectively with massive unlabeled datasets via
joint reconstruction objectives, and utilizing diverse multimodal data for
enhanced representation robustness. Experiments demonstrate that MoCa
consistently improves performance across MMEB and ViDoRe-v2 benchmarks,
achieving new state-of-the-art results, and exhibits strong scalability with
both model size and training data on MMEB.
68634959588cea0da970c8a5
Haonan Chen
claimed_verified
2025-07-01T10:13:35.715Z
66add675c7a575aa0e03d5f3
/avatars/b72b18130664c1de197c1f8df371aa70.svg
Haonan Chen
user
Haon-Chen
68634959588cea0da970c8a6
Hong Liu
claimed_verified
2025-07-01T10:13:40.207Z
648ba0d68e7f7a927675d4a3
/avatars/d82ced93656e03d60c8b55010694f908.svg
Hong Liu
user
hongliu9903
68634959588cea0da970c8a7
Yuping Luo
claimed_verified
2025-07-01T10:13:37.881Z
64ed68ce60f6345da7014b38
/avatars/adfc156482ef5570dc69329aa53975e6.svg
Yuping Luo
user
roosephu
68634959588cea0da970c8a8
Liang Wang
68634959588cea0da970c8a9
Nan Yang
68634959588cea0da970c8aa
Furu Wei
68634959588cea0da970c8ab
Zhicheng Dou
MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional
  Multimodal Embeddings
Multimodal embedding models, built upon causal Vision Language Models (VLMs),
have shown promise in various tasks. However, current approaches face three key
limitations: the use of causal attention in VLM backbones is suboptimal for
embedding tasks; scalability issues due to reliance on high-quality labeled
paired data for contrastive learning; and limited diversity in training
objectives and data. To address these issues, we propose MoCa, a two-stage
framework for transforming pre-trained VLMs into effective bidirectional
multimodal embedding models. The first stage, Modality-aware Continual
Pre-training, introduces a joint reconstruction objective that simultaneously
denoises interleaved text and image inputs, enhancing bidirectional
context-aware reasoning. The second stage, Heterogeneous Contrastive
Fine-tuning, leverages diverse, semantically rich multimodal data beyond simple
image-caption pairs to enhance generalization and alignment. Our method
addresses the stated limitations by introducing bidirectional attention through
continual pre-training, scaling effectively with massive unlabeled datasets via
joint reconstruction objectives, and utilizing diverse multimodal data for
enhanced representation robustness. Experiments demonstrate that MoCa
consistently improves performance across MMEB and ViDoRe-v2 benchmarks,
achieving new state-of-the-art results, and exhibits strong scalability with
both model size and training data on MMEB.
MoCa, a two-stage framework, enhances pre-trained VLMs into effective bidirectional multimodal embedding models by addressing limitations through modality-aware continual pre-training and heterogeneous contrastive fine-tuning.
causal Vision Language Models bidirectional multimodal embedding models causal attention contrastive learning joint reconstruction objective bidirectional attention continual pre-training heterogeneous contrastive fine-tuning multimodal data MMEB ViDoRe-v2