MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents
AI agents with advanced reasoning and tool use capabilities have demonstrated
impressive performance in web browsing for deep search. While existing
benchmarks such as BrowseComp evaluate these browsing abilities, they primarily
focus on textual information, overlooking the prevalence of multimodal content.
To bridge this gap, we introduce MM-BrowseComp, a novel benchmark comprising
224 challenging, hand-crafted questions specifically designed to assess agents'
multimodal retrieval and reasoning capabilities. These questions often
incorporate images in prompts, and crucial information encountered during the
search and reasoning process may also be embedded within images or videos on
webpages. Consequently, methods relying solely on text prove insufficient for
our benchmark. Additionally, we provide a verified checklist for each question,
enabling fine-grained analysis of multimodal dependencies and reasoning paths.
Our comprehensive evaluation of state-of-the-art models on MM-BrowseComp
reveals that even top models like OpenAI o3 with tools achieve only 29.02\%
accuracy, highlighting the suboptimal multimodal capabilities and lack of
native multimodal reasoning in current models.
68a5b381c4b3ea17d2d2cfbc
Shilong Li
68a5b381c4b3ea17d2d2cfbd
Xingyuan Bu
admin_assigned
2025-08-20T19:40:08.260Z
6444e7765691ca69b0d95856
https://cdn-avatars.huggingface.co/v1/production/uploads/6444e7765691ca69b0d95856/WbtUXVbC2VIribtVAsLp7.jpeg
Xingyuan Bu
user
sefira32
68a5b381c4b3ea17d2d2cfbe
Wenjie Wang
68a5b381c4b3ea17d2d2cfbf
Jiaheng Liu
68a5b381c4b3ea17d2d2cfc0
Jun Dong
68a5b381c4b3ea17d2d2cfc1
Haoyang He
68a5b381c4b3ea17d2d2cfc2
Hao Lu
68a5b381c4b3ea17d2d2cfc3
Haozhe Zhang
68a5b381c4b3ea17d2d2cfc4
Chenchen Jing
68a5b381c4b3ea17d2d2cfc5
Zhen Li
68a5b381c4b3ea17d2d2cfc6
Chuanhao Li
68a5b381c4b3ea17d2d2cfc7
Jiayi Tian
68a5b381c4b3ea17d2d2cfc8
Chenchen Zhang
68a5b381c4b3ea17d2d2cfc9
Tianhao Peng
68a5b381c4b3ea17d2d2cfca
Yancheng He
68a5b381c4b3ea17d2d2cfcb
Jihao Gu
68a5b381c4b3ea17d2d2cfcc
Yuanxing Zhang
68a5b381c4b3ea17d2d2cfcd
Jian Yang
68a5b381c4b3ea17d2d2cfce
Ge Zhang
claimed_verified
2025-08-25T19:31:59.606Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
68a5b381c4b3ea17d2d2cfcf
Wenhao Huang
68a5b381c4b3ea17d2d2cfd0
Wangchunshu Zhou
68a5b381c4b3ea17d2d2cfd1
Zhaoxiang Zhang
68a5b381c4b3ea17d2d2cfd2
Ruizhe Ding
68a5b381c4b3ea17d2d2cfd3
Shilei Wen
MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents
AI agents with advanced reasoning and tool use capabilities have demonstrated
impressive performance in web browsing for deep search. While existing
benchmarks such as BrowseComp evaluate these browsing abilities, they primarily
focus on textual information, overlooking the prevalence of multimodal content.
To bridge this gap, we introduce MM-BrowseComp, a novel benchmark comprising
224 challenging, hand-crafted questions specifically designed to assess agents'
multimodal retrieval and reasoning capabilities. These questions often
incorporate images in prompts, and crucial information encountered during the
search and reasoning process may also be embedded within images or videos on
webpages. Consequently, methods relying solely on text prove insufficient for
our benchmark. Additionally, we provide a verified checklist for each question,
enabling fine-grained analysis of multimodal dependencies and reasoning paths.
Our comprehensive evaluation of state-of-the-art models on MM-BrowseComp
reveals that even top models like OpenAI o3 with tools achieve only 29.02\%
accuracy, highlighting the suboptimal multimodal capabilities and lack of
native multimodal reasoning in current models.
A new benchmark, MM-BrowseComp, evaluates AI agents' multimodal retrieval and reasoning capabilities, revealing limitations in current models' handling of images and videos in web browsing tasks.
multimodal content multimodal retrieval multimodal reasoning MM-BrowseComp OpenAI o3