MIBench: Evaluating Multimodal Large Language Models over Multiple
  Images
Built on the power of LLMs, numerous multimodal large language models (MLLMs)
have recently achieved remarkable performance on various vision-language tasks
across multiple benchmarks. However, most existing MLLMs and benchmarks
primarily focus on single-image input scenarios, leaving the performance of
MLLMs when handling realistic multiple images remain underexplored. Although a
few benchmarks consider multiple images, their evaluation dimensions and
samples are very limited. Therefore, in this paper, we propose a new benchmark
MIBench, to comprehensively evaluate fine-grained abilities of MLLMs in
multi-image scenarios. Specifically, MIBench categorizes the multi-image
abilities into three scenarios: multi-image instruction (MII), multimodal
knowledge-seeking (MKS) and multimodal in-context learning (MIC), and
constructs 13 tasks with a total of 13K annotated samples. During data
construction, for MII and MKS, we extract correct options from manual
annotations and create challenging distractors to obtain multiple-choice
questions. For MIC, to enable an in-depth evaluation, we set four sub-tasks and
transform the original datasets into in-context learning formats. We evaluate
several open-source MLLMs and close-source MLLMs on the proposed MIBench. The
results reveal that although current models excel in single-image tasks, they
exhibit significant shortcomings when faced with multi-image inputs, such as
confused fine-grained perception, limited multi-image reasoning, and unstable
in-context learning. The annotated data in MIBench is available at
https://huggingface.co/datasets/StarBottle/MIBench.
669f9709cf9bff621ccbb2f0
Haowei Liu
669f9709cf9bff621ccbb2f1
Xi Zhang
669f9709cf9bff621ccbb2f2
Haiyang Xu
claimed_verified
2024-07-23T11:46:38.953Z
645b10e80c73ea27d13f7aca
/avatars/95e565306472a15067440b5b43e07a6f.svg
xuhaiyang
user
xhyandwyy
669f9709cf9bff621ccbb2f3
Yaya Shi
admin_assigned
2024-07-23T12:02:48.710Z
63172db0106740aaad4f9954
/avatars/50f3056c4ba45eed3bbdaced0e1c360f.svg
Yaya Shi
user
ShiYaya
669f9709cf9bff621ccbb2f4
Chaoya Jiang
admin_assigned
2024-07-23T12:03:02.787Z
61b9976efd429dff1ed3bd44
/avatars/5cdaa04e970e1e3dcfbb38ba89c0660a.svg
jiangchaoya
user
jcy
669f9709cf9bff621ccbb2f5
Ming Yan
claimed_verified
2024-08-14T08:11:48.287Z
64771cfdd7cf39f2e9381aa9
/avatars/48adf00c3b653df02628f80511639e19.svg
Ming
user
MingYan123
669f9709cf9bff621ccbb2f6
Ji Zhang
669f9709cf9bff621ccbb2f7
Fei Huang
admin_assigned
2024-07-23T12:03:49.360Z
635b8b6a37c6a2c12e2cce00
/avatars/229fb72180529141515d1df797b33709.svg
Fei Huang
user
hzhwcmhf
669f9709cf9bff621ccbb2f8
Chunfeng Yuan
669f9709cf9bff621ccbb2f9
Bing Li
669f9709cf9bff621ccbb2fa
Weiming Hu
MIBench: Evaluating Multimodal Large Language Models over Multiple
  Images
Built on the power of LLMs, numerous multimodal large language models (MLLMs)
have recently achieved remarkable performance on various vision-language tasks
across multiple benchmarks. However, most existing MLLMs and benchmarks
primarily focus on single-image input scenarios, leaving the performance of
MLLMs when handling realistic multiple images remain underexplored. Although a
few benchmarks consider multiple images, their evaluation dimensions and
samples are very limited. Therefore, in this paper, we propose a new benchmark
MIBench, to comprehensively evaluate fine-grained abilities of MLLMs in
multi-image scenarios. Specifically, MIBench categorizes the multi-image
abilities into three scenarios: multi-image instruction (MII), multimodal
knowledge-seeking (MKS) and multimodal in-context learning (MIC), and
constructs 13 tasks with a total of 13K annotated samples. During data
construction, for MII and MKS, we extract correct options from manual
annotations and create challenging distractors to obtain multiple-choice
questions. For MIC, to enable an in-depth evaluation, we set four sub-tasks and
transform the original datasets into in-context learning formats. We evaluate
several open-source MLLMs and close-source MLLMs on the proposed MIBench. The
results reveal that although current models excel in single-image tasks, they
exhibit significant shortcomings when faced with multi-image inputs, such as
confused fine-grained perception, limited multi-image reasoning, and unstable
in-context learning. The annotated data in MIBench is available at
https://huggingface.co/datasets/StarBottle/MIBench.
MIBench is a new benchmark that evaluates MLLMs on multi-image tasks, revealing their shortcomings in multi-image reasoning and fine-grained perception.
LLMs MLLMs MIBench multi-image instruction multimodal knowledge-seeking multimodal in-context learning