MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation
  in Videos
Multimodal Language Language Models (MLLMs) demonstrate the emerging
abilities of "world models" -- interpreting and reasoning about complex
real-world dynamics. To assess these abilities, we posit videos are the ideal
medium, as they encapsulate rich representations of real-world dynamics and
causalities. To this end, we introduce MMWorld, a new benchmark for
multi-discipline, multi-faceted multimodal video understanding. MMWorld
distinguishes itself from previous video understanding benchmarks with two
unique advantages: (1) multi-discipline, covering various disciplines that
often require domain expertise for comprehensive understanding; (2)
multi-faceted reasoning, including explanation, counterfactual thinking, future
prediction, etc. MMWorld consists of a human-annotated dataset to evaluate
MLLMs with questions about the whole videos and a synthetic dataset to analyze
MLLMs within a single modality of perception. Together, MMWorld encompasses
1,910 videos across seven broad disciplines and 69 subdisciplines, complete
with 6,627 question-answer pairs and associated captions. The evaluation
includes 2 proprietary and 10 open-source MLLMs, which struggle on MMWorld
(e.g., GPT-4V performs the best with only 52.3\% accuracy), showing large room
for improvement. Further ablation studies reveal other interesting findings
such as models' different skill sets from humans. We hope MMWorld can serve as
an essential step towards world model evaluation in videos.
666a574b5c3bb7e4a6c7ff51
Xuehai He
admin_assigned
2024-06-13T20:09:58.196Z
6310493158d83e8f64dc8c55
/avatars/5f91ac4dfec0d6a5bf7bad6094f0fd0f.svg
Xuehai He
user
Xuehai
666a574b5c3bb7e4a6c7ff52
Weixi Feng
admin_assigned
2024-06-13T20:10:04.923Z
6259e9489326c60347dd2b6c
/avatars/6aae6282ddee96b899b7384154e9b0a0.svg
Weixi Feng
user
weixifeng
666a574b5c3bb7e4a6c7ff53
Kaizhi Zheng
666a574b5c3bb7e4a6c7ff54
Yujie Lu
admin_assigned
2024-06-13T20:10:19.029Z
633cfb6ad5935998f7582b7b
https://cdn-avatars.huggingface.co/v1/production/uploads/633cfb6ad5935998f7582b7b/mzU4lI_GWPsYOFZHTBsYA.jpeg
Yujie Lu
user
FunCube
666a574b5c3bb7e4a6c7ff55
Wanrong Zhu
admin_assigned
2024-06-13T20:10:25.249Z
633a7e91c8eb7b089034fb3c
/avatars/0e03d80aa469de131cc17bb0c5c016b9.svg
Wanrong Zhu
user
VegB
666a574b5c3bb7e4a6c7ff56
Jiachen Li
claimed_verified
2024-06-13T09:31:10.205Z
640d7ec5fdeaae13907fc488
/avatars/acf43ea155105a51c8612dacc4725091.svg
Jiachen Li
user
jiachenli-ucsb
666a574b5c3bb7e4a6c7ff57
Yue Fan
claimed_verified
2024-06-30T20:58:09.562Z
656b8a62e8bf55919a6a2d61
/avatars/26b75683dff73c51f966e9c15f04fd9f.svg
Yue Fan
user
yfan1997
666a574b5c3bb7e4a6c7ff58
Jianfeng Wang
admin_assigned
2024-06-13T20:10:55.977Z
641404e74e5305c14f269b3c
/avatars/0dbe862cde7e4bef25bedf860584d8c1.svg
Jianfeng Wang
user
amsword8
666a574b5c3bb7e4a6c7ff59
Linjie Li
admin_assigned
2024-06-13T20:11:02.085Z
63db16fff03c3d71ef397206
/avatars/bfb7e0d730b7d03302799d5d2828d97d.svg
Linjie Li
user
linjieli222
666a574b5c3bb7e4a6c7ff5a
Zhengyuan Yang
admin_assigned
2024-06-13T20:11:08.514Z
630713411801ecc7d2592a7c
/avatars/fb36f69f03421c3a2a7f72ba0858fa60.svg
Zhengyuan Yang
user
zyang39
666a574b5c3bb7e4a6c7ff5b
Kevin Lin
admin_assigned
2024-06-13T20:11:21.085Z
6298fd95b58e71e2ac9f3ad8
/avatars/7d34644d537bc5c17cf1e4ce4095355c.svg
Kevin Lin
user
kevinlin311tw
666a574b5c3bb7e4a6c7ff5c
William Yang Wang
admin_assigned
2024-06-13T20:11:27.471Z
6476607efb22e3b77f3f1452
/avatars/22f40c81e6b6e09bdb66a5afd978f98b.svg
William Yang Wang
user
wangwilliamyang
666a574b5c3bb7e4a6c7ff5d
Lijuan Wang
admin_assigned
2024-06-13T20:11:32.768Z
6413521d4e5305c14f22e110
/avatars/a6f8d0573e678f79bc3c0b7897b818ce.svg
Lijuan Wang
user
Lijuan
666a574b5c3bb7e4a6c7ff5e
Xin Eric Wang
claimed_verified
2024-06-13T09:31:12.418Z
64679a226192d39142245e5e
/avatars/05abee0b6317f100923936ca2099e9eb.svg
Xin Eric Wang
user
xw-eric
MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation
  in Videos
Multimodal Language Language Models (MLLMs) demonstrate the emerging
abilities of "world models" -- interpreting and reasoning about complex
real-world dynamics. To assess these abilities, we posit videos are the ideal
medium, as they encapsulate rich representations of real-world dynamics and
causalities. To this end, we introduce MMWorld, a new benchmark for
multi-discipline, multi-faceted multimodal video understanding. MMWorld
distinguishes itself from previous video understanding benchmarks with two
unique advantages: (1) multi-discipline, covering various disciplines that
often require domain expertise for comprehensive understanding; (2)
multi-faceted reasoning, including explanation, counterfactual thinking, future
prediction, etc. MMWorld consists of a human-annotated dataset to evaluate
MLLMs with questions about the whole videos and a synthetic dataset to analyze
MLLMs within a single modality of perception. Together, MMWorld encompasses
1,910 videos across seven broad disciplines and 69 subdisciplines, complete
with 6,627 question-answer pairs and associated captions. The evaluation
includes 2 proprietary and 10 open-source MLLMs, which struggle on MMWorld
(e.g., GPT-4V performs the best with only 52.3\% accuracy), showing large room
for improvement. Further ablation studies reveal other interesting findings
such as models' different skill sets from humans. We hope MMWorld can serve as
an essential step towards world model evaluation in videos.
MMWorld is a new benchmark for evaluating multimodal video understanding in various disciplines and reasoning tasks, highlighting gaps between current models and human performance.
multimodal language language models world models video understanding benchmark multi-discipline multi-faceted reasoning human-annotated dataset synthetic dataset question-answer pairs GPT-4V