CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding
  Benchmark
As the capabilities of large multimodal models (LMMs) continue to advance,
evaluating the performance of LMMs emerges as an increasing need. Additionally,
there is an even larger gap in evaluating the advanced knowledge and reasoning
abilities of LMMs in non-English contexts such as Chinese. We introduce CMMMU,
a new Chinese Massive Multi-discipline Multimodal Understanding benchmark
designed to evaluate LMMs on tasks demanding college-level subject knowledge
and deliberate reasoning in a Chinese context. CMMMU is inspired by and
strictly follows the annotation and analysis pattern of MMMU.
  CMMMU includes 12k manually collected multimodal questions from college
exams, quizzes, and textbooks, covering six core disciplines: Art & Design,
Business, Science, Health & Medicine, Humanities & Social Science, and Tech &
Engineering, like its companion, MMMU. These questions span 30 subjects and
comprise 39 highly heterogeneous image types, such as charts, diagrams, maps,
tables, music sheets, and chemical structures.
  CMMMU focuses on complex perception and reasoning with domain-specific
knowledge in the Chinese context. We evaluate 11 open-source LLMs and one
proprietary GPT-4V(ision). Even GPT-4V only achieves accuracies of 42%,
indicating a large space for improvement. CMMMU will boost the community to
build the next-generation LMMs towards expert artificial intelligence and
promote the democratization of LMMs by providing diverse language contexts.
65af321777381188bd7548fd
Ge Zhang
claimed_verified
2024-01-23T09:29:10.355Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
65af321777381188bd7548fe
Xinrun Du
65af321777381188bd7548ff
Bei Chen
claimed_verified
2024-01-23T13:49:00.403Z
64cc65d786d8dc0caa6ab3cd
/avatars/23dc126f7332825e713807fb1c000ab6.svg
Bei Chen
user
BellaBei
65af321777381188bd754900
Yiming Liang
65af321777381188bd754901
Tongxu Luo
claimed_verified
2024-05-27T08:49:33.842Z
6421b07e918f0fd889f0a682
/avatars/314b55c2428426c846d9449f98db4355.svg
Tongxu Luo
user
Zeno-Luo
65af321777381188bd754902
Tianyu Zheng
claimed_verified
2024-02-23T16:36:09.819Z
64ab99dcb76bfd863eba64c1
https://cdn-avatars.huggingface.co/v1/production/uploads/64ab99dcb76bfd863eba64c1/UBXwDPx17X-gl-SzBPvrc.jpeg
TY.Zheng
user
aaabiao
65af321777381188bd754903
Kang Zhu
claimed_verified
2024-10-18T08:55:24.980Z
6578265ddea7e2122d02f6ba
https://cdn-avatars.huggingface.co/v1/production/uploads/6578265ddea7e2122d02f6ba/Bh6JjoVF5ceLSjV7Z7nTk.jpeg
kang zhu
user
kangz
65af321777381188bd754904
Yuyang Cheng
65af321777381188bd754905
Chunpu Xu
admin_assigned
2024-01-23T08:27:34.925Z
6580333b64bab467b9bcb615
/avatars/753afae2fefc748791550c733aa2c80b.svg
Chunpu Xu
user
cpxu
65af321777381188bd754906
Shuyue Guo
65af321777381188bd754907
Haoran Zhang
65af321777381188bd754908
Xingwei Qu
65af321777381188bd754909
Junjie Wang
claimed_verified
2024-01-23T08:09:03.576Z
62579c55b98dcaa7e0de285d
https://cdn-avatars.huggingface.co/v1/production/uploads/62579c55b98dcaa7e0de285d/0YUd5nloul_bW9yolDGGo.jpeg
wangjunjie
user
wanng
65af321777381188bd75490a
Ruibin Yuan
admin_assigned
2024-01-23T08:29:41.058Z
5fd6f670053c8345eddc1b68
https://cdn-avatars.huggingface.co/v1/production/uploads/5fd6f670053c8345eddc1b68/cuTsu2krRYHC6zYGD2dpQ.jpeg
Ruibin Yuan
user
a43992899
65af321777381188bd75490b
Yizhi Li
claimed_verified
2024-05-31T06:54:24.108Z
6382252f54421460665ec501
https://cdn-avatars.huggingface.co/v1/production/uploads/6382252f54421460665ec501/gW9fev3T5QPcNq4f9hqB1.jpeg
Yizhi Li
user
yizhilll
65af321777381188bd75490c
Zekun Wang
claimed_verified
2024-10-07T07:32:59.886Z
6149a9e95347647e6bb68882
https://cdn-avatars.huggingface.co/v1/production/uploads/6149a9e95347647e6bb68882/Jddln1FxScCeVgTSCNBpr.png
Zekun Moore Wang
user
ZenMoore
65af321777381188bd75490d
Yudong Liu
65af321777381188bd75490e
Yu-Hsuan Tsai
65af321777381188bd75490f
Fengji Zhang
admin_assigned
2024-01-23T08:30:52.592Z
64c8d4ac120a85440bfad55d
/avatars/d65334288d69700250de51d0df344d9b.svg
Fengji Zhang
user
zfj1998
65af321777381188bd754910
Chenghua Lin
admin_assigned
2024-01-23T08:30:59.767Z
6442ba123610a28a4ad5382f
/avatars/97f2d3d2c10058ea41d7bf9087e1f619.svg
Chenghua Lin
user
chenghualin
65af321777381188bd754911
Wenhao Huang
claimed_verified
2024-10-13T20:32:10.735Z
65bb11cb00a03997849e9e85
/avatars/17022b0254192a837f4fe00d84389cda.svg
Wenhao Huang
user
StephenHuang
65af321777381188bd754912
Wenhu Chen
admin_assigned
2024-01-23T08:31:14.767Z
6313a86154e6e5d9f0f94e04
https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
Wenhu Chen
user
wenhu
65af321777381188bd754913
Jie Fu
admin_assigned
2024-01-23T08:32:02.954Z
641a6895fb5ffff5ac79d593
https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/vxvwsto3llOEWGqQKGMYx.jpeg
Jie Fu
user
bigaidream
CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding
  Benchmark
As the capabilities of large multimodal models (LMMs) continue to advance,
evaluating the performance of LMMs emerges as an increasing need. Additionally,
there is an even larger gap in evaluating the advanced knowledge and reasoning
abilities of LMMs in non-English contexts such as Chinese. We introduce CMMMU,
a new Chinese Massive Multi-discipline Multimodal Understanding benchmark
designed to evaluate LMMs on tasks demanding college-level subject knowledge
and deliberate reasoning in a Chinese context. CMMMU is inspired by and
strictly follows the annotation and analysis pattern of MMMU.
  CMMMU includes 12k manually collected multimodal questions from college
exams, quizzes, and textbooks, covering six core disciplines: Art & Design,
Business, Science, Health & Medicine, Humanities & Social Science, and Tech &
Engineering, like its companion, MMMU. These questions span 30 subjects and
comprise 39 highly heterogeneous image types, such as charts, diagrams, maps,
tables, music sheets, and chemical structures.
  CMMMU focuses on complex perception and reasoning with domain-specific
knowledge in the Chinese context. We evaluate 11 open-source LLMs and one
proprietary GPT-4V(ision). Even GPT-4V only achieves accuracies of 42%,
indicating a large space for improvement. CMMMU will boost the community to
build the next-generation LMMs towards expert artificial intelligence and
promote the democratization of LMMs by providing diverse language contexts.
CMMMU evaluates large multimodal models in college-level Chinese tasks across six disciplines, highlighting gaps and promoting expert-level artificial intelligence development.
large multimodal models CMMMU MMMU complex perception reasoning domain-specific knowledge expert artificial intelligence