Benchmarking Multi-Image Understanding in Vision and Language Models:
  Perception, Knowledge, Reasoning, and Multi-Hop Reasoning
The advancement of large language models (LLMs) has significantly broadened
the scope of applications in natural language processing, with multi-modal LLMs
extending these capabilities to integrate and interpret visual data. However,
existing benchmarks for visual language models (VLMs) predominantly focus on
single-image inputs, neglecting the crucial aspect of multi-image
understanding. In this paper, we introduce a Multi-Image Relational Benchmark
MIRB, designed to evaluate VLMs' ability to compare, analyze, and reason across
multiple images. Our benchmark encompasses four categories: perception, visual
world knowledge, reasoning, and multi-hop reasoning. Through a comprehensive
evaluation of a wide range of open-source and closed-source models, we
demonstrate that while open-source VLMs were shown to approach the performance
of GPT-4V in single-image tasks, a significant performance gap remains in
multi-image reasoning tasks. Our findings also reveal that even the
state-of-the-art GPT-4V model struggles with our benchmark, underscoring the
need for further research and development in this area. We believe our
contribution of MIRB could serve as a testbed for developing the
next-generation multi-modal models.
66727eeef022d14aa15540ea
Bingchen Zhao
claimed_verified
2024-06-19T07:37:20.887Z
62dcd71075e9787ec5aa41ba
/avatars/f37ce036b76180ed0fa004f9c8c09363.svg
Bingchen Zhao
user
tennant
66727eeef022d14aa15540eb
Yongshuo Zong
admin_assigned
2024-06-19T08:58:49.432Z
643d5de2c17bfb2256f14b4b
/avatars/1124128ce69ae6868fd1a2c06420d72f.svg
Yongshuo Zong
user
ys-zong
66727eeef022d14aa15540ec
Letian Zhang
admin_assigned
2024-06-19T08:58:56.282Z
648efcad5adc64d0d982ff42
/avatars/2697fa23b576ce2f93ec2107723dbcf9.svg
Letian Zhang
user
Letian2003
66727eeef022d14aa15540ed
Timothy Hospedales
Benchmarking Multi-Image Understanding in Vision and Language Models:
  Perception, Knowledge, Reasoning, and Multi-Hop Reasoning
The advancement of large language models (LLMs) has significantly broadened
the scope of applications in natural language processing, with multi-modal LLMs
extending these capabilities to integrate and interpret visual data. However,
existing benchmarks for visual language models (VLMs) predominantly focus on
single-image inputs, neglecting the crucial aspect of multi-image
understanding. In this paper, we introduce a Multi-Image Relational Benchmark
MIRB, designed to evaluate VLMs' ability to compare, analyze, and reason across
multiple images. Our benchmark encompasses four categories: perception, visual
world knowledge, reasoning, and multi-hop reasoning. Through a comprehensive
evaluation of a wide range of open-source and closed-source models, we
demonstrate that while open-source VLMs were shown to approach the performance
of GPT-4V in single-image tasks, a significant performance gap remains in
multi-image reasoning tasks. Our findings also reveal that even the
state-of-the-art GPT-4V model struggles with our benchmark, underscoring the
need for further research and development in this area. We believe our
contribution of MIRB could serve as a testbed for developing the
next-generation multi-modal models.
A benchmark for visual language models (VLMs) evaluating their performance across multiple images, highlighting gaps in multi-image reasoning.
large language models multi-modal LLMs visual language models VLMs single-image inputs multi-image understanding Multi-Image Relational Benchmark MIRB perception visual world knowledge reasoning multi-hop reasoning GPT-4V multi-modal models