M3Retrieve: Benchmarking Multimodal Retrieval for Medicine
With the increasing use of RetrievalAugmented Generation (RAG), strong
retrieval models have become more important than ever. In healthcare,
multimodal retrieval models that combine information from both text and images
offer major advantages for many downstream tasks such as question answering,
cross-modal retrieval, and multimodal summarization, since medical data often
includes both formats. However, there is currently no standard benchmark to
evaluate how well these models perform in medical settings. To address this
gap, we introduce M3Retrieve, a Multimodal Medical Retrieval Benchmark.
M3Retrieve, spans 5 domains,16 medical fields, and 4 distinct tasks, with over
1.2 Million text documents and 164K multimodal queries, all collected under
approved licenses. We evaluate leading multimodal retrieval models on this
benchmark to explore the challenges specific to different medical specialities
and to understand their impact on retrieval performance. By releasing
M3Retrieve, we aim to enable systematic evaluation, foster model innovation,
and accelerate research toward building more capable and reliable multimodal
retrieval systems for medical applications. The dataset and the baselines code
are available in this github page https://github.com/AkashGhosh/M3Retrieve.
68e79e2f7ae125f9582e6c8c
Arkadeep Acharya
68e79e2f7ae125f9582e6c8d
Akash Ghosh
claimed_verified
2025-10-09T12:47:52.935Z
630e850c3fc17ffc50f752e5
/avatars/43e4aee228323211b8cf55b783cc54d2.svg
Akash Ghosh
user
Agcs12
68e79e2f7ae125f9582e6c8e
Pradeepika Verma
68e79e2f7ae125f9582e6c8f
Kitsuchart Pasupa
68e79e2f7ae125f9582e6c90
Sriparna Saha
68e79e2f7ae125f9582e6c91
Priti Singh
M3Retrieve: Benchmarking Multimodal Retrieval for Medicine
With the increasing use of RetrievalAugmented Generation (RAG), strong
retrieval models have become more important than ever. In healthcare,
multimodal retrieval models that combine information from both text and images
offer major advantages for many downstream tasks such as question answering,
cross-modal retrieval, and multimodal summarization, since medical data often
includes both formats. However, there is currently no standard benchmark to
evaluate how well these models perform in medical settings. To address this
gap, we introduce M3Retrieve, a Multimodal Medical Retrieval Benchmark.
M3Retrieve, spans 5 domains,16 medical fields, and 4 distinct tasks, with over
1.2 Million text documents and 164K multimodal queries, all collected under
approved licenses. We evaluate leading multimodal retrieval models on this
benchmark to explore the challenges specific to different medical specialities
and to understand their impact on retrieval performance. By releasing
M3Retrieve, we aim to enable systematic evaluation, foster model innovation,
and accelerate research toward building more capable and reliable multimodal
retrieval systems for medical applications. The dataset and the baselines code
are available in this github page https://github.com/AkashGhosh/M3Retrieve.
M3Retrieve is a new benchmark for evaluating multimodal retrieval models in healthcare, covering multiple domains and tasks with a large dataset of text and image data.
RetrievalAugmented Generation multimodal retrieval models question answering cross-modal retrieval multimodal summarization M3Retrieve Multimodal Medical Retrieval Benchmark