DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Search
Multimodal Large Language Models (MLLMs) in real-world applications require
access to external knowledge sources and must remain responsive to the dynamic
and ever-changing real-world information in order to address
information-seeking and knowledge-intensive user queries. Existing approaches,
such as retrieval augmented generation (RAG) methods, search agents, and search
equipped MLLMs, often suffer from rigid pipelines, excessive search calls, and
poorly constructed search queries, which result in inefficiencies and
suboptimal outcomes. To address these limitations, we present DeepMMSearch-R1,
the first multimodal LLM capable of performing on-demand, multi-turn web
searches and dynamically crafting queries for both image and text search tools.
Specifically, DeepMMSearch-R1 can initiate web searches based on relevant crops
of the input image making the image search more effective, and can iteratively
adapt text search queries based on retrieved information, thereby enabling
self-reflection and self-correction. Our approach relies on a two-stage
training pipeline: a cold start supervised finetuning phase followed by an
online reinforcement learning optimization. For training, we introduce
DeepMMSearchVQA, a novel multimodal VQA dataset created through an automated
pipeline intermixed with real-world information from web search tools. This
dataset contains diverse, multi-hop queries that integrate textual and visual
information, teaching the model when to search, what to search for, which
search tool to use and how to reason over the retrieved information. We conduct
extensive experiments across a range of knowledge-intensive benchmarks to
demonstrate the superiority of our approach. Finally, we analyze the results
and provide insights that are valuable for advancing multimodal web-search.
68eefd3a486b78128f0e3361
Kartik Narayan
claimed_verified
2025-10-15T17:03:48.916Z
65f92935ee1763d31ae255c1
/avatars/94c6b36e5c42b60813d3b557e3947fa7.svg
Kartik Narayan
user
kartiknarayan
68eefd3a486b78128f0e3362
Yang Xu
68eefd3a486b78128f0e3363
Tian Cao
68eefd3a486b78128f0e3364
Kavya Nerella
68eefd3a486b78128f0e3365
Vishal M. Patel
68eefd3a486b78128f0e3366
Navid Shiee
68eefd3a486b78128f0e3367
Peter Grasch
68eefd3a486b78128f0e3368
Chao Jia
68eefd3a486b78128f0e3369
Yinfei Yang
68eefd3a486b78128f0e336a
Zhe Gan
DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Search
Multimodal Large Language Models (MLLMs) in real-world applications require
access to external knowledge sources and must remain responsive to the dynamic
and ever-changing real-world information in order to address
information-seeking and knowledge-intensive user queries. Existing approaches,
such as retrieval augmented generation (RAG) methods, search agents, and search
equipped MLLMs, often suffer from rigid pipelines, excessive search calls, and
poorly constructed search queries, which result in inefficiencies and
suboptimal outcomes. To address these limitations, we present DeepMMSearch-R1,
the first multimodal LLM capable of performing on-demand, multi-turn web
searches and dynamically crafting queries for both image and text search tools.
Specifically, DeepMMSearch-R1 can initiate web searches based on relevant crops
of the input image making the image search more effective, and can iteratively
adapt text search queries based on retrieved information, thereby enabling
self-reflection and self-correction. Our approach relies on a two-stage
training pipeline: a cold start supervised finetuning phase followed by an
online reinforcement learning optimization. For training, we introduce
DeepMMSearchVQA, a novel multimodal VQA dataset created through an automated
pipeline intermixed with real-world information from web search tools. This
dataset contains diverse, multi-hop queries that integrate textual and visual
information, teaching the model when to search, what to search for, which
search tool to use and how to reason over the retrieved information. We conduct
extensive experiments across a range of knowledge-intensive benchmarks to
demonstrate the superiority of our approach. Finally, we analyze the results
and provide insights that are valuable for advancing multimodal web-search.