Visual Haystacks: Answering Harder Questions About Sets of Images
Recent advancements in Large Multimodal Models (LMMs) have made significant
progress in the field of single-image visual question answering. However, these
models face substantial challenges when tasked with queries that span extensive
collections of images, similar to real-world scenarios like searching through
large photo albums, finding specific information across the internet, or
monitoring environmental changes through satellite imagery. This paper explores
the task of Multi-Image Visual Question Answering (MIQA): given a large set of
images and a natural language query, the task is to generate a relevant and
grounded response. We propose a new public benchmark, dubbed "Visual Haystacks
(VHs)," specifically designed to evaluate LMMs' capabilities in visual
retrieval and reasoning over sets of unrelated images, where we perform
comprehensive evaluations demonstrating that even robust closed-source models
struggle significantly. Towards addressing these shortcomings, we introduce
MIRAGE (Multi-Image Retrieval Augmented Generation), a novel retrieval/QA
framework tailored for LMMs that confronts the challenges of MIQA with marked
efficiency and accuracy improvements over baseline methods. Our evaluation
shows that MIRAGE surpasses closed-source GPT-4o models by up to 11% on the VHs
benchmark and offers up to 3.4x improvements in efficiency over text-focused
multi-stage approaches.
669fc8149c7cc392d3a19489
Tsung-Han Wu
claimed_verified
2025-04-18T09:42:10.796Z
644a767044b75fd95805d232
https://cdn-avatars.huggingface.co/v1/production/uploads/644a767044b75fd95805d232/vHA2vI_B3CpXapdBEwspB.jpeg
Patrick (Tsung-Han) Wu
user
tsunghanwu
669fc8149c7cc392d3a1948a
Giscard Biamby
669fc8149c7cc392d3a1948b
Jerome Quenum
669fc8149c7cc392d3a1948c
Ritwik Gupta
claimed_verified
2025-05-28T12:31:23.940Z
630489e5dae2eb7d083e78b1
https://cdn-avatars.huggingface.co/v1/production/uploads/630489e5dae2eb7d083e78b1/RKjwiquU1JzaHoV4op42F.jpeg
Ritwik Gupta
user
RitwikGupta
669fc8149c7cc392d3a1948d
Joseph E. Gonzalez
669fc8149c7cc392d3a1948e
Trevor Darrell
669fc8149c7cc392d3a1948f
David M. Chan
claimed_verified
2024-07-23T15:51:46.294Z
6388f68c43d8b0797a09ff84
https://cdn-avatars.huggingface.co/v1/production/uploads/1669920369989-noauth.jpeg
David Chan
user
davidchan
Visual Haystacks: Answering Harder Questions About Sets of Images
Recent advancements in Large Multimodal Models (LMMs) have made significant
progress in the field of single-image visual question answering. However, these
models face substantial challenges when tasked with queries that span extensive
collections of images, similar to real-world scenarios like searching through
large photo albums, finding specific information across the internet, or
monitoring environmental changes through satellite imagery. This paper explores
the task of Multi-Image Visual Question Answering (MIQA): given a large set of
images and a natural language query, the task is to generate a relevant and
grounded response. We propose a new public benchmark, dubbed "Visual Haystacks
(VHs)," specifically designed to evaluate LMMs' capabilities in visual
retrieval and reasoning over sets of unrelated images, where we perform
comprehensive evaluations demonstrating that even robust closed-source models
struggle significantly. Towards addressing these shortcomings, we introduce
MIRAGE (Multi-Image Retrieval Augmented Generation), a novel retrieval/QA
framework tailored for LMMs that confronts the challenges of MIQA with marked
efficiency and accuracy improvements over baseline methods. Our evaluation
shows that MIRAGE surpasses closed-source GPT-4o models by up to 11% on the VHs
benchmark and offers up to 3.4x improvements in efficiency over text-focused
multi-stage approaches.
MIRAGE, a novel retrieval/QA framework, significantly improves the performance and efficiency of Multi-Image Visual Question Answering tasks compared to existing models on the Visual Haystacks benchmark.
Large Multimodal Models Multi-Image Visual Question Answering Visual Haystacks MIRAGE retrieval QA framework visual retrieval visual reasoning efficiency improvements GPT-4o