Needle In A Multimodal Haystack
With the rapid advancement of multimodal large language models (MLLMs), their
evaluation has become increasingly comprehensive. However, understanding long
multimodal content, as a foundational ability for real-world applications,
remains underexplored. In this work, we present Needle In A Multimodal Haystack
(MM-NIAH), the first benchmark specifically designed to systematically evaluate
the capability of existing MLLMs to comprehend long multimodal documents. Our
benchmark includes three types of evaluation tasks: multimodal retrieval,
counting, and reasoning. In each task, the model is required to answer the
questions according to different key information scattered throughout the given
multimodal document. Evaluating the leading MLLMs on MM-NIAH, we observe that
existing models still have significant room for improvement on these tasks,
especially on vision-centric evaluation. We hope this work can provide a
platform for further research on long multimodal document comprehension and
contribute to the advancement of MLLMs. Code and benchmark are released at
https://github.com/OpenGVLab/MM-NIAH.
6669285a759649b9b0a567c5
Weiyun Wang
admin_assigned
2024-06-17T13:52:35.011Z
619507e7b74b6c591f794340
https://cdn-avatars.huggingface.co/v1/production/uploads/619507e7b74b6c591f794340/JbPDoy6Ko1V1-6oJJwFV8.jpeg
Weiyun Wang
user
Weiyun1025
6669285a759649b9b0a567c6
Shuibo Zhang
claimed_verified
2025-03-21T11:50:50.497Z
666ff0e9251af54f960e9870
/avatars/c6b727821e29f9d40afdf0ef6935cd0b.svg
Shuibai Zhang
user
Shuibai12138
6669285a759649b9b0a567c7
Yiming Ren
admin_assigned
2024-06-17T13:52:50.031Z
659f996896e2038d1aa2a77c
/avatars/6f6204f9d9a7aca63c2b4599b3065977.svg
yiming ren
user
Flash128
6669285a759649b9b0a567c8
Yuchen Duan
claimed_verified
2024-06-17T13:43:24.553Z
6495a4d6d1cec0e2c2c96cdd
/avatars/e8b42301514094c8af6fa1e5fcb53119.svg
Duan Yuchen
user
duanyuchen
6669285a759649b9b0a567c9
Tiantong Li
admin_assigned
2024-06-17T13:52:55.709Z
645efb6fb5c9a8666d105401
/avatars/39d6c51cd1f15319b6c4efe9c72f1df2.svg
Tiantong Li
user
casually5101
6669285a759649b9b0a567ca
Shuo Liu
6669285a759649b9b0a567cb
Mengkang Hu
claimed_verified
2025-02-19T09:07:29.396Z
6237df4a5ab9df625fb70c1a
https://cdn-avatars.huggingface.co/v1/production/uploads/6237df4a5ab9df625fb70c1a/qVM-7ww4ozUDu_MS4JivM.jpeg
Mengkang Hu
user
MengkangHu
6669285a759649b9b0a567cc
Zhe Chen
claimed_verified
2024-12-09T14:49:33.003Z
64119264f0f81eb569e0d569
/avatars/00dccedea51ac7d2f398ca9d65a8e78b.svg
Zhe Chen
user
czczup
6669285a759649b9b0a567cd
Kaipeng Zhang
claimed_verified
2024-06-17T07:33:57.728Z
65f1713552c38a91e0a445e8
/avatars/47ab3ada51c9b9976ac1cd0c4301c373.svg
kaipeng
user
kpzhang996
6669285a759649b9b0a567ce
Lewei Lu
admin_assigned
2024-06-17T13:53:50.340Z
65ead3ea908526a39082e641
/avatars/dcf870695fd56b06ca03d82f831e9019.svg
Lewei Lu
user
luotto
6669285a759649b9b0a567cf
Xizhou Zhu
admin_assigned
2024-06-17T13:53:56.048Z
64ae2359179421d320b1694b
/avatars/c387a75191005bcaa473091de5383a10.svg
Xizhou Zhu
user
Einsiedler
6669285a759649b9b0a567d0
Ping Luo
admin_assigned
2024-06-17T13:54:07.526Z
666bd450d528e9911600b302
/avatars/30fd6b03b70af8018e8d5f99b89c68a5.svg
luoping1121
user
luoping
6669285a759649b9b0a567d1
Yu Qiao
6669285a759649b9b0a567d2
Jifeng Dai
6669285a759649b9b0a567d3
Wenqi Shao
admin_assigned
2024-06-17T13:54:17.560Z
64b3fd42eec33e27dcc4c941
/avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
Wenqi Shao
user
wqshao126
6669285a759649b9b0a567d4
Wenhai Wang
admin_assigned
2024-06-17T13:54:39.048Z
64d1c560c0c627dfa71bdbe0
/avatars/f42794fe25bffcd870a1bcee69b95298.svg
wenhai.wang
user
wangwhcore
Needle In A Multimodal Haystack
With the rapid advancement of multimodal large language models (MLLMs), their
evaluation has become increasingly comprehensive. However, understanding long
multimodal content, as a foundational ability for real-world applications,
remains underexplored. In this work, we present Needle In A Multimodal Haystack
(MM-NIAH), the first benchmark specifically designed to systematically evaluate
the capability of existing MLLMs to comprehend long multimodal documents. Our
benchmark includes three types of evaluation tasks: multimodal retrieval,
counting, and reasoning. In each task, the model is required to answer the
questions according to different key information scattered throughout the given
multimodal document. Evaluating the leading MLLMs on MM-NIAH, we observe that
existing models still have significant room for improvement on these tasks,
especially on vision-centric evaluation. We hope this work can provide a
platform for further research on long multimodal document comprehension and
contribute to the advancement of MLLMs. Code and benchmark are released at
https://github.com/OpenGVLab/MM-NIAH.
NN-NIAH is a benchmark that evaluates MLLMs' comprehension of long multimodal documents across retrieval, counting, and reasoning tasks.
multimodal large language models MLLMs Needle In A Multimodal Haystack NN-NIAH multimodal retrieval multimodal document comprehension vision-centric evaluation