MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for
  Reasoning-Intensive Multimodal Retrieval
We introduce MRMR, the first expert-level multidisciplinary multimodal
retrieval benchmark requiring intensive reasoning. MRMR contains 1,502 queries
spanning 23 domains, with positive documents carefully verified by human
experts. Compared to prior benchmarks, MRMR introduces three key advancements.
First, it challenges retrieval systems across diverse areas of expertise,
enabling fine-grained model comparison across domains. Second, queries are
reasoning-intensive, with images requiring deeper interpretation such as
diagnosing microscopic slides. We further introduce Contradiction Retrieval, a
novel task requiring models to identify conflicting concepts. Finally, queries
and documents are constructed as image-text interleaved sequences. Unlike
earlier benchmarks restricted to single images or unimodal documents, MRMR
offers a realistic setting with multi-image queries and mixed-modality corpus
documents. We conduct an extensive evaluation of 4 categories of multimodal
retrieval systems and 14 frontier models on MRMR. The text embedding model
Qwen3-Embedding with LLM-generated image captions achieves the highest
performance, highlighting substantial room for improving multimodal retrieval
models. Although latest multimodal models such as Ops-MM-Embedding perform
competitively on expert-domain queries, they fall short on reasoning-intensive
tasks. We believe that MRMR paves the way for advancing multimodal retrieval in
more realistic and challenging scenarios.
68ec7396cd07fb414898ca08
Siyue Zhang
68ec7396cd07fb414898ca09
Yuan Gao
claimed_verified
2025-10-13T10:05:22.025Z
6720a6d52890caca00aae9e7
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/A8h2DRemFRQit6NRAhkSv.png
Yuan Gao
user
ItzYog
68ec7396cd07fb414898ca0a
Xiao Zhou
68ec7396cd07fb414898ca0b
Yilun Zhao
68ec7396cd07fb414898ca0c
Tingyu Song
claimed_verified
2025-10-22T15:46:39.147Z
64dc29d9b5d625e0e9a6ecb9
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/QxGBsnk1cNsBEPqSx4ae-.jpeg
Tingyu Song
user
songtingyu
68ec7396cd07fb414898ca0d
Arman Cohan
68ec7396cd07fb414898ca0e
Anh Tuan Luu
68ec7396cd07fb414898ca0f
Chen Zhao
MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for
  Reasoning-Intensive Multimodal Retrieval
We introduce MRMR, the first expert-level multidisciplinary multimodal
retrieval benchmark requiring intensive reasoning. MRMR contains 1,502 queries
spanning 23 domains, with positive documents carefully verified by human
experts. Compared to prior benchmarks, MRMR introduces three key advancements.
First, it challenges retrieval systems across diverse areas of expertise,
enabling fine-grained model comparison across domains. Second, queries are
reasoning-intensive, with images requiring deeper interpretation such as
diagnosing microscopic slides. We further introduce Contradiction Retrieval, a
novel task requiring models to identify conflicting concepts. Finally, queries
and documents are constructed as image-text interleaved sequences. Unlike
earlier benchmarks restricted to single images or unimodal documents, MRMR
offers a realistic setting with multi-image queries and mixed-modality corpus
documents. We conduct an extensive evaluation of 4 categories of multimodal
retrieval systems and 14 frontier models on MRMR. The text embedding model
Qwen3-Embedding with LLM-generated image captions achieves the highest
performance, highlighting substantial room for improving multimodal retrieval
models. Although latest multimodal models such as Ops-MM-Embedding perform
competitively on expert-domain queries, they fall short on reasoning-intensive
tasks. We believe that MRMR paves the way for advancing multimodal retrieval in
more realistic and challenging scenarios.
MRMR is a benchmark for expert-level multidisciplinary multimodal retrieval that includes reasoning-intensive tasks, contradiction retrieval, and image-text interleaved sequences, highlighting the need for improved multimodal models.
multimodal retrieval expert-level multidisciplinary reasoning-intensive Contradiction Retrieval image-text interleaved multimodal models text embedding LLM-generated image captions Ops-MM-Embedding