ModernVBERT: Towards Smaller Visual Document Retrievers
Multimodal embedding models are gaining prevalence, notably for document
retrieval as efficient alternatives to text-only pipelines. These models are
typically built by finetuning large vision-language decoders (VLMs) with
contrastive losses on text-image pairs. In this work, we show that, while
cost-efficient, this repurposing approach often bottlenecks retrieval
performance. Through controlled experiments, we establish a principled recipe
for improving visual document retrieval models. We notably measure the impact
of attention masking, image resolution, modality alignment data regimes, and
late interaction centered contrastive objectives which emerge as central
performance factors. Building on these insights, we release ModernVBERT, a
compact 250M-parameter vision-language encoder that outperforms models up to 10
times larger when finetuned on document retrieval tasks. Models and code are
made available at https://huggingface.co/ModernVBERT.
68de63af70ada21878c75026
Paul Teiletche
68de63af70ada21878c75027
Quentin Macé
claimed_verified
2025-10-03T12:52:04.668Z
661e945eebe3616a1b09e279
https://cdn-avatars.huggingface.co/v1/production/uploads/661e945eebe3616a1b09e279/U3DL1BNouUpcusCKAPZm0.jpeg
Quentin Macé
user
QuentinJG
68de63af70ada21878c75028
Max Conti
68de63af70ada21878c75029
Antonio Loison
claimed_verified
2025-10-11T14:05:49.301Z
6377b63b24d97f9f7ec70064
https://cdn-avatars.huggingface.co/v1/production/uploads/6377b63b24d97f9f7ec70064/vBl68PAwwp353MOeOahMH.jpeg
Antonio Loison
user
antonioloison
68de63af70ada21878c7502a
Gautier Viaud
68de63af70ada21878c7502b
Pierre Colombo
68de63af70ada21878c7502c
Manuel Faysse
claimed_verified
2025-10-03T12:52:47.596Z
60f2e021adf471cbdf8bb660
https://cdn-avatars.huggingface.co/v1/production/uploads/1654090481550-60f2e021adf471cbdf8bb660.jpeg
Manuel Faysse
user
manu
ModernVBERT: Towards Smaller Visual Document Retrievers
Multimodal embedding models are gaining prevalence, notably for document
retrieval as efficient alternatives to text-only pipelines. These models are
typically built by finetuning large vision-language decoders (VLMs) with
contrastive losses on text-image pairs. In this work, we show that, while
cost-efficient, this repurposing approach often bottlenecks retrieval
performance. Through controlled experiments, we establish a principled recipe
for improving visual document retrieval models. We notably measure the impact
of attention masking, image resolution, modality alignment data regimes, and
late interaction centered contrastive objectives which emerge as central
performance factors. Building on these insights, we release ModernVBERT, a
compact 250M-parameter vision-language encoder that outperforms models up to 10
times larger when finetuned on document retrieval tasks. Models and code are
made available at https://huggingface.co/ModernVBERT.
ModernVBERT, a compact vision-language encoder, outperforms larger models in document retrieval by optimizing attention masking, image resolution, modality alignment, and contrastive objectives.
multimodal embedding models document retrieval vision-language decoders contrastive losses attention masking image resolution modality alignment contrastive objectives vision-language encoder