GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards
  General Medical AI
Large Vision-Language Models (LVLMs) are capable of handling diverse data
types such as imaging, text, and physiological signals, and can be applied in
various fields. In the medical field, LVLMs have a high potential to offer
substantial assistance for diagnosis and treatment. Before that, it is crucial
to develop benchmarks to evaluate LVLMs' effectiveness in various medical
applications. Current benchmarks are often built upon specific academic
literature, mainly focusing on a single domain, and lacking varying perceptual
granularities. Thus, they face specific challenges, including limited clinical
relevance, incomplete evaluations, and insufficient guidance for interactive
LVLMs. To address these limitations, we developed the GMAI-MMBench, the most
comprehensive general medical AI benchmark with well-categorized data structure
and multi-perceptual granularity to date. It is constructed from 285 datasets
across 39 medical image modalities, 18 clinical-related tasks, 18 departments,
and 4 perceptual granularities in a Visual Question Answering (VQA) format.
Additionally, we implemented a lexical tree structure that allows users to
customize evaluation tasks, accommodating various assessment needs and
substantially supporting medical AI research and applications. We evaluated 50
LVLMs, and the results show that even the advanced GPT-4o only achieves an
accuracy of 52%, indicating significant room for improvement. Moreover, we
identified five key insufficiencies in current cutting-edge LVLMs that need to
be addressed to advance the development of better medical applications. We
believe that GMAI-MMBench will stimulate the community to build the next
generation of LVLMs toward GMAI.
  Project Page: https://uni-medical.github.io/GMAI-MMBench.github.io/
66b57e63e1f790d2510cd344
Pengcheng Chen
66b57e63e1f790d2510cd345
Jin Ye
claimed_verified
2024-11-25T20:39:46.118Z
656292f6412dcdebf2c1c6a8
/avatars/310604f8c7d701565e9a659e8e85621d.svg
Jin Ye
user
EugeneYonng
66b57e63e1f790d2510cd346
Guoan Wang
claimed_verified
2024-08-09T07:47:30.273Z
6409b7eb96aae649740e2920
https://cdn-avatars.huggingface.co/v1/production/uploads/1678358480751-noauth.png
Guoan Wang
user
SolooooMan
66b57e63e1f790d2510cd347
Yanjun Li
66b57e63e1f790d2510cd348
Zhongying Deng
66b57e63e1f790d2510cd349
Wei Li
66b57e63e1f790d2510cd34a
Tianbin Li
admin_assigned
2024-08-09T09:34:10.951Z
64aff8ecf808e8d2023804ce
/avatars/13d32347626814801c0f9b2d893b6741.svg
li
user
tianbin
66b57e63e1f790d2510cd34b
Haodong Duan
admin_assigned
2024-08-09T09:29:22.932Z
63ee1379190ddd6214efd73a
https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
HAODONG DUAN
user
KennyUTC
66b57e63e1f790d2510cd34c
Ziyan Huang
admin_assigned
2024-08-09T09:29:29.120Z
643fddd6388f31f64ccb9344
/avatars/c029b59cfbbc4124489e72fa8e71ee9a.svg
Ziyan Huang
user
ziyanhuang
66b57e63e1f790d2510cd34d
Yanzhou Su
66b57e63e1f790d2510cd34e
Benyou Wang
admin_assigned
2024-08-09T09:29:45.299Z
637c6703ca8542a0ba900ccb
/avatars/288ed63a1efa566c3f01e850c6ba5dd5.svg
Wang
user
Benyou
66b57e63e1f790d2510cd34f
Shaoting Zhang
66b57e63e1f790d2510cd350
Bin Fu
66b57e63e1f790d2510cd351
Jianfei Cai
admin_assigned
2024-08-09T09:30:13.163Z
656fda082f058b368c27d1b9
/avatars/cba33f6b20fc93ac9d3d579cbd7f839a.svg
cai jianfei
user
genify
66b57e63e1f790d2510cd352
Bohan Zhuang
admin_assigned
2024-08-09T09:30:19.255Z
65a8a828ef14f9e6037dfad2
/avatars/55fdc0393b2e37ac998ddbdb1e1ff636.svg
Bohan Zhuang
user
BohanZ
66b57e63e1f790d2510cd353
Eric J Seibel
66b57e63e1f790d2510cd354
Junjun He
admin_assigned
2024-08-09T09:30:32.614Z
66b593026cef22e6ba6adb8a
/avatars/8a94d55b85177e84d65dd0bd537e335f.svg
JunjunHe
user
JunjunHe
66b57e63e1f790d2510cd355
Yu Qiao
GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards
  General Medical AI
Large Vision-Language Models (LVLMs) are capable of handling diverse data
types such as imaging, text, and physiological signals, and can be applied in
various fields. In the medical field, LVLMs have a high potential to offer
substantial assistance for diagnosis and treatment. Before that, it is crucial
to develop benchmarks to evaluate LVLMs' effectiveness in various medical
applications. Current benchmarks are often built upon specific academic
literature, mainly focusing on a single domain, and lacking varying perceptual
granularities. Thus, they face specific challenges, including limited clinical
relevance, incomplete evaluations, and insufficient guidance for interactive
LVLMs. To address these limitations, we developed the GMAI-MMBench, the most
comprehensive general medical AI benchmark with well-categorized data structure
and multi-perceptual granularity to date. It is constructed from 285 datasets
across 39 medical image modalities, 18 clinical-related tasks, 18 departments,
and 4 perceptual granularities in a Visual Question Answering (VQA) format.
Additionally, we implemented a lexical tree structure that allows users to
customize evaluation tasks, accommodating various assessment needs and
substantially supporting medical AI research and applications. We evaluated 50
LVLMs, and the results show that even the advanced GPT-4o only achieves an
accuracy of 52%, indicating significant room for improvement. Moreover, we
identified five key insufficiencies in current cutting-edge LVLMs that need to
be addressed to advance the development of better medical applications. We
believe that GMAI-MMBench will stimulate the community to build the next
generation of LVLMs toward GMAI.
  Project Page: https://uni-medical.github.io/GMAI-MMBench.github.io/
GMAI-MMBench is a comprehensive general medical AI benchmark that evaluates large vision-language models across multiple medical domains and perceptual granularities, identifying key insufficiencies and highlighting areas for improvement.
Large Vision-Language Models LVLMs medical image modalities clinical-related tasks perceptual granularities Visual Question Answering VQA lexical tree structure GPT-4 GMAI