M3Ret: Unleashing Zero-shot Multimodal Medical Image Retrieval via
  Self-Supervision
Medical image retrieval is essential for clinical decision-making and
translational research, relying on discriminative visual representations. Yet,
current methods remain fragmented, relying on separate architectures and
training strategies for 2D, 3D, and video-based medical data. This
modality-specific design hampers scalability and inhibits the development of
unified representations. To enable unified learning, we curate a large-scale
hybrid-modality dataset comprising 867,653 medical imaging samples, including
2D X-rays and ultrasounds, RGB endoscopy videos, and 3D CT scans. Leveraging
this dataset, we train M3Ret, a unified visual encoder without any
modality-specific customization. It successfully learns transferable
representations using both generative (MAE) and contrastive (SimDINO)
self-supervised learning (SSL) paradigms. Our approach sets a new
state-of-the-art in zero-shot image-to-image retrieval across all individual
modalities, surpassing strong baselines such as DINOv3 and the text-supervised
BMC-CLIP. More remarkably, strong cross-modal alignment emerges without paired
data, and the model generalizes to unseen MRI tasks, despite never observing
MRI during pretraining, demonstrating the generalizability of purely visual
self-supervision to unseen modalities. Comprehensive analyses further validate
the scalability of our framework across model and data sizes. These findings
deliver a promising signal to the medical imaging community, positioning M3Ret
as a step toward foundation models for visual SSL in multimodal medical image
understanding.
68b7bc15295f15ff6091134b
Che Liu
68b7bc15295f15ff6091134c
Zheng Jiang
68b7bc15295f15ff6091134d
Chengyu Fang
68b7bc15295f15ff6091134e
Heng Guo
68b7bc15295f15ff6091134f
Yan-Jie Zhou
68b7bc15295f15ff60911350
Jiaqi Qu
68b7bc15295f15ff60911351
Le Lu
68b7bc15295f15ff60911352
Minfeng Xu
M3Ret: Unleashing Zero-shot Multimodal Medical Image Retrieval via
  Self-Supervision
Medical image retrieval is essential for clinical decision-making and
translational research, relying on discriminative visual representations. Yet,
current methods remain fragmented, relying on separate architectures and
training strategies for 2D, 3D, and video-based medical data. This
modality-specific design hampers scalability and inhibits the development of
unified representations. To enable unified learning, we curate a large-scale
hybrid-modality dataset comprising 867,653 medical imaging samples, including
2D X-rays and ultrasounds, RGB endoscopy videos, and 3D CT scans. Leveraging
this dataset, we train M3Ret, a unified visual encoder without any
modality-specific customization. It successfully learns transferable
representations using both generative (MAE) and contrastive (SimDINO)
self-supervised learning (SSL) paradigms. Our approach sets a new
state-of-the-art in zero-shot image-to-image retrieval across all individual
modalities, surpassing strong baselines such as DINOv3 and the text-supervised
BMC-CLIP. More remarkably, strong cross-modal alignment emerges without paired
data, and the model generalizes to unseen MRI tasks, despite never observing
MRI during pretraining, demonstrating the generalizability of purely visual
self-supervision to unseen modalities. Comprehensive analyses further validate
the scalability of our framework across model and data sizes. These findings
deliver a promising signal to the medical imaging community, positioning M3Ret
as a step toward foundation models for visual SSL in multimodal medical image
understanding.
M3Ret, a unified visual encoder trained on a large-scale hybrid-modality dataset, achieves state-of-the-art zero-shot image-to-image retrieval and cross-modal alignment using generative and contrastive self-supervised learning paradigms.
visual encoder self-supervised learning SSL generative contrastive MAE SimDINO zero-shot image-to-image retrieval cross-modal alignment foundation models multimodal medical image understanding