MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution
  Real-World Scenarios that are Difficult for Humans?
Comprehensive evaluation of Multimodal Large Language Models (MLLMs) has
recently garnered widespread attention in the research community. However, we
observe that existing benchmarks present several common barriers that make it
difficult to measure the significant challenges that models face in the real
world, including: 1) small data scale leads to a large performance variance; 2)
reliance on model-based annotations results in restricted data quality; 3)
insufficient task difficulty, especially caused by the limited image
resolution. To tackle these issues, we introduce MME-RealWorld. Specifically,
we collect more than 300K images from public datasets and the Internet,
filtering 13,366 high-quality images for annotation. This involves the
efforts of professional 25 annotators and 7 experts in MLLMs, contributing
to 29,429 question-answer pairs that cover 43 subtasks across 5
real-world scenarios, extremely challenging even for humans. As far as we know,
MME-RealWorld is the largest manually annotated benchmark to date, featuring
the highest resolution and a targeted focus on real-world applications. We
further conduct a thorough evaluation involving 28 prominent MLLMs, such as
GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet. Our results show that even the
most advanced models struggle with our benchmarks, where none of them reach
60% accuracy. The challenges of perceiving high-resolution images and
understanding complex real-world scenarios remain urgent issues to be
addressed. The data and evaluation code are released at
https://mme-realworld.github.io/ .
66cbf6dab395ddc37b56e9c8
Yi-Fan Zhang
claimed_verified
2024-08-27T07:43:53.881Z
623d8ca4c29adf5ef6175615
https://cdn-avatars.huggingface.co/v1/production/uploads/623d8ca4c29adf5ef6175615/q7lHao7UPwU1u7YLSP56m.jpeg
Yi-Fan Zhang
user
yifanzhang114
66cbf6dab395ddc37b56e9c9
Huanyu Zhang
claimed_verified
2024-08-26T07:14:54.194Z
65e816bbcfd12cd15b052a0e
/avatars/4d92da469afdba8cd7dc645b98236011.svg
Huanyu_Zhang
user
huanyu112
66cbf6dab395ddc37b56e9ca
Haochen Tian
claimed_verified
2024-11-24T22:07:22.152Z
6448baa3e780dbfc89058bc3
/avatars/44c243c8eb4714cdcdaa7cd04e5a9716.svg
Micheal Tian
user
StarBurger
66cbf6dab395ddc37b56e9cb
Chaoyou Fu
66cbf6dab395ddc37b56e9cc
Shuangqing Zhang
66cbf6dab395ddc37b56e9cd
Junfei Wu
claimed_verified
2025-06-12T12:46:14.335Z
65d801194c28f74f7f5e8059
/avatars/f18622ba64e4c19c1288ccdc1761cf4b.svg
Junfei Wu
user
Hyperwjf
66cbf6dab395ddc37b56e9ce
Feng Li
66cbf6dab395ddc37b56e9cf
Kun Wang
66cbf6dab395ddc37b56e9d0
Qingsong Wen
66cbf6dab395ddc37b56e9d1
Zhang Zhang
66cbf6dab395ddc37b56e9d2
Liang Wang
66cbf6dab395ddc37b56e9d3
Rong Jin
66cbf6dab395ddc37b56e9d4
Tieniu Tan
MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution
  Real-World Scenarios that are Difficult for Humans?
Comprehensive evaluation of Multimodal Large Language Models (MLLMs) has
recently garnered widespread attention in the research community. However, we
observe that existing benchmarks present several common barriers that make it
difficult to measure the significant challenges that models face in the real
world, including: 1) small data scale leads to a large performance variance; 2)
reliance on model-based annotations results in restricted data quality; 3)
insufficient task difficulty, especially caused by the limited image
resolution. To tackle these issues, we introduce MME-RealWorld. Specifically,
we collect more than 300K images from public datasets and the Internet,
filtering 13,366 high-quality images for annotation. This involves the
efforts of professional 25 annotators and 7 experts in MLLMs, contributing
to 29,429 question-answer pairs that cover 43 subtasks across 5
real-world scenarios, extremely challenging even for humans. As far as we know,
MME-RealWorld is the largest manually annotated benchmark to date, featuring
the highest resolution and a targeted focus on real-world applications. We
further conduct a thorough evaluation involving 28 prominent MLLMs, such as
GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet. Our results show that even the
most advanced models struggle with our benchmarks, where none of them reach
60% accuracy. The challenges of perceiving high-resolution images and
understanding complex real-world scenarios remain urgent issues to be
addressed. The data and evaluation code are released at
https://mme-realworld.github.io/ .
MME-RealWorld is a large manually annotated benchmark featuring high-resolution images for evaluating the performance of Multimodal Large Language Models in real-world scenarios.
Multimodal Large Language Models MME-RealWorld image resolution real-world scenarios question-answer pairs high-resolution images evaluation benchmarks