SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with
  Text-Rich Visual Comprehension
Comprehending text-rich visual content is paramount for the practical
application of Multimodal Large Language Models (MLLMs), since text-rich
scenarios are ubiquitous in the real world, which are characterized by the
presence of extensive texts embedded within images. Recently, the advent of
MLLMs with impressive versatility has raised the bar for what we can expect
from MLLMs. However, their proficiency in text-rich scenarios has yet to be
comprehensively and objectively assessed, since current MLLM benchmarks
primarily focus on evaluating general visual comprehension. In this work, we
introduce SEED-Bench-2-Plus, a benchmark specifically designed for evaluating
text-rich visual comprehension of MLLMs. Our benchmark comprises 2.3K
multiple-choice questions with precise human annotations, spanning three broad
categories: Charts, Maps, and Webs, each of which covers a wide spectrum of
text-rich scenarios in the real world. These categories, due to their inherent
complexity and diversity, effectively simulate real-world text-rich
environments. We further conduct a thorough evaluation involving 34 prominent
MLLMs (including GPT-4V, Gemini-Pro-Vision and Claude-3-Opus) and emphasize the
current limitations of MLLMs in text-rich visual comprehension. We hope that
our work can serve as a valuable addition to existing MLLM benchmarks,
providing insightful observations and inspiring further research in the area of
text-rich visual comprehension with MLLMs. The dataset and evaluation code can
be accessed at https://github.com/AILab-CVC/SEED-Bench.
662b12f3c90d81a396fa5b75
Bohao Li
admin_assigned
2024-04-26T12:03:44.583Z
6310b7e70a43f97f6c56191e
/avatars/4a24c76e34d12c3d6230a4a081115f72.svg
Bohao Li
user
BreakLee
662b12f3c90d81a396fa5b76
Yuying Ge
admin_assigned
2024-04-26T12:03:56.492Z
6455cc8f654d8bccae50e4d4
/avatars/506a9992e5bf52e06d37cc22e4b307c0.svg
Yuying Ge
user
tttoaster
662b12f3c90d81a396fa5b77
Yi Chen
662b12f3c90d81a396fa5b78
Yixiao Ge
admin_assigned
2024-04-26T12:04:23.679Z
640e9762b03f4cd29f58d982
/avatars/81da37d628163fe3e094b247c7c3a3b5.svg
Yixiao Ge
user
yxgeee
662b12f3c90d81a396fa5b79
Ruimao Zhang
admin_assigned
2024-04-26T12:04:35.029Z
65324848220f490dbaa3b8c9
/avatars/cec7dcadf4657bb4f61e2c9a8075521b.svg
Zhang
user
Ruimao
662b12f3c90d81a396fa5b7a
Ying Shan
SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with
  Text-Rich Visual Comprehension
Comprehending text-rich visual content is paramount for the practical
application of Multimodal Large Language Models (MLLMs), since text-rich
scenarios are ubiquitous in the real world, which are characterized by the
presence of extensive texts embedded within images. Recently, the advent of
MLLMs with impressive versatility has raised the bar for what we can expect
from MLLMs. However, their proficiency in text-rich scenarios has yet to be
comprehensively and objectively assessed, since current MLLM benchmarks
primarily focus on evaluating general visual comprehension. In this work, we
introduce SEED-Bench-2-Plus, a benchmark specifically designed for evaluating
text-rich visual comprehension of MLLMs. Our benchmark comprises 2.3K
multiple-choice questions with precise human annotations, spanning three broad
categories: Charts, Maps, and Webs, each of which covers a wide spectrum of
text-rich scenarios in the real world. These categories, due to their inherent
complexity and diversity, effectively simulate real-world text-rich
environments. We further conduct a thorough evaluation involving 34 prominent
MLLMs (including GPT-4V, Gemini-Pro-Vision and Claude-3-Opus) and emphasize the
current limitations of MLLMs in text-rich visual comprehension. We hope that
our work can serve as a valuable addition to existing MLLM benchmarks,
providing insightful observations and inspiring further research in the area of
text-rich visual comprehension with MLLMs. The dataset and evaluation code can
be accessed at https://github.com/AILab-CVC/SEED-Bench.
A benchmark evaluating text-rich visual comprehension for Multimodal Large Language Models (MLLMs) introduces SEED-Bench-2-Plus with comprehensive human annotations across diverse scenarios.
Multimodal Large Language Models MLLMs text-rich visual comprehension SEED-Bench-2-Plus multiple-choice questions human annotations general visual comprehension GPT-4V Gemini-Pro-Vision Claude-3-Opus