MMVU: Measuring Expert-Level Multi-Discipline Video Understanding
We introduce MMVU, a comprehensive expert-level, multi-discipline benchmark
for evaluating foundation models in video understanding. MMVU includes 3,000
expert-annotated questions spanning 27 subjects across four core disciplines:
Science, Healthcare, Humanities & Social Sciences, and Engineering. Compared to
prior benchmarks, MMVU features three key advancements. First, it challenges
models to apply domain-specific knowledge and perform expert-level reasoning to
analyze specialized-domain videos, moving beyond the basic visual perception
typically assessed in current video benchmarks. Second, each example is
annotated by human experts from scratch. We implement strict data quality
controls to ensure the high quality of the dataset. Finally, each example is
enriched with expert-annotated reasoning rationals and relevant domain
knowledge, facilitating in-depth analysis. We conduct an extensive evaluation
of 32 frontier multimodal foundation models on MMVU. The latest
System-2-capable models, o1 and Gemini 2.0 Flash Thinking, achieve the highest
performance among the tested models. However, they still fall short of matching
human expertise. Through in-depth error analyses and case studies, we offer
actionable insights for future advancements in expert-level,
knowledge-intensive video understanding for specialized domains.
67906f432565fc5140d72dc3
Yilun Zhao
claimed_verified
2025-01-22T20:49:53.483Z
62f662bcc58915315c4eccea
https://cdn-avatars.huggingface.co/v1/production/uploads/62f662bcc58915315c4eccea/zOAQLONfMP88zr70sxHK-.jpeg
Yilun Zhao
user
yilunzhao
67906f432565fc5140d72dc4
Lujing Xie
admin_assigned
2025-01-22T14:06:45.496Z
64ffd83b96ec8a52185dfb54
/avatars/a4fadc7e2f5c1125d5d455de4d5c9b8e.svg
Lujing Xie
user
leeroylucas
67906f432565fc5140d72dc5
Haowei Zhang
claimed_verified
2025-01-22T10:59:50.465Z
637169557a5e5d8efdc3e58e
https://cdn-avatars.huggingface.co/v1/production/uploads/1668515232215-637169557a5e5d8efdc3e58e.jpeg
Haowei Zhang
user
freesky
67906f432565fc5140d72dc6
Guo Gan
claimed_verified
2025-03-07T09:10:43.494Z
65dfeee3d16fb170031df293
https://cdn-avatars.huggingface.co/v1/production/uploads/65dfeee3d16fb170031df293/DZefH96Ol6joeAaGVBe3j.jpeg
gan
user
guo9
67906f432565fc5140d72dc7
Yitao Long
67906f432565fc5140d72dc8
Zhiyuan Hu
admin_assigned
2025-01-22T14:07:09.079Z
65e9945f6bcbcae600b7e64f
/avatars/8aa986a6c0e35c55d5d1461d1dc11ac3.svg
Zhiyuan Hu
user
zhiyhu
67906f432565fc5140d72dc9
Tongyan Hu
claimed_verified
2025-01-22T10:59:48.493Z
66e83ec5deb449d8d856e78d
/avatars/c5e56be65fcacb3192ce10ba6d8f48e2.svg
Tongyan Hu
user
entropyhu
67906f432565fc5140d72dca
Weiyuan Chen
claimed_verified
2025-01-22T14:04:09.444Z
6652e84bff6ccc0ef5ac7055
/avatars/70a8b6a0bae5f9e954fa15f56ab2ddc3.svg
Weiyuan Chen
user
WeiyuanChen01
67906f432565fc5140d72dcb
Chuhan Li
admin_assigned
2025-01-22T14:07:16.582Z
65415f1d5168c4f3487a2103
https://cdn-avatars.huggingface.co/v1/production/uploads/65415f1d5168c4f3487a2103/qJuzDpOGSDL4E1-L2lGWW.jpeg
Chuhan Li
user
ChuhanLi
67906f432565fc5140d72dcc
Junyang Song
claimed_verified
2025-02-11T15:34:00.857Z
67909add9551780939a950f9
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/vS8OfOtpYr0nNXWixn1Er.png
Junyang Song
user
UndetectedAtom
67906f432565fc5140d72dcd
Zhijian Xu
67906f432565fc5140d72dce
Chengye Wang
67906f432565fc5140d72dcf
Weifeng Pan
claimed_verified
2025-01-22T10:08:18.785Z
67907fdb6146e0f96241dcc3
/avatars/ba16cb87dce1e3ccbbe8091a3fe553fc.svg
Wf Pan
user
Phil-01
67906f432565fc5140d72dd0
Ziyao Shangguan
admin_assigned
2025-01-22T14:06:39.157Z
65dea56779f827c045b1df96
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Jtx4JcM5CNgoVyzxTC93S.jpeg
Ziyao Shangguan
user
ziyaosg
67906f432565fc5140d72dd1
Xiangru Tang
admin_assigned
2025-01-22T14:07:56.034Z
63357c608adfa81faf2ac180
/avatars/ae0314c644f882251baf59b9134fd36f.svg
Xiangru Tang
user
RTT1
67906f432565fc5140d72dd2
Zhenwen Liang
admin_assigned
2025-01-22T14:08:01.981Z
62ffa3f8311cad266f9af236
/avatars/203dac40bc546ee25a01d8715a4b3049.svg
Zhenwen Liang
user
invokerliang
67906f432565fc5140d72dd3
Yixin Liu
67906f432565fc5140d72dd4
Chen Zhao
67906f432565fc5140d72dd5
Arman Cohan
admin_assigned
2025-01-22T14:09:26.876Z
5f5ba21188f57f65f951f255
https://cdn-avatars.huggingface.co/v1/production/uploads/1599840760465-noauth.png
Arman Cohan
user
armanc
MMVU: Measuring Expert-Level Multi-Discipline Video Understanding
We introduce MMVU, a comprehensive expert-level, multi-discipline benchmark
for evaluating foundation models in video understanding. MMVU includes 3,000
expert-annotated questions spanning 27 subjects across four core disciplines:
Science, Healthcare, Humanities & Social Sciences, and Engineering. Compared to
prior benchmarks, MMVU features three key advancements. First, it challenges
models to apply domain-specific knowledge and perform expert-level reasoning to
analyze specialized-domain videos, moving beyond the basic visual perception
typically assessed in current video benchmarks. Second, each example is
annotated by human experts from scratch. We implement strict data quality
controls to ensure the high quality of the dataset. Finally, each example is
enriched with expert-annotated reasoning rationals and relevant domain
knowledge, facilitating in-depth analysis. We conduct an extensive evaluation
of 32 frontier multimodal foundation models on MMVU. The latest
System-2-capable models, o1 and Gemini 2.0 Flash Thinking, achieve the highest
performance among the tested models. However, they still fall short of matching
human expertise. Through in-depth error analyses and case studies, we offer
actionable insights for future advancements in expert-level,
knowledge-intensive video understanding for specialized domains.
A comprehensive benchmark for evaluating foundation models in video understanding includes domain-specific questions and expert evaluations, highlighting the gap between current models and human expertise.
multimodal foundation models expert-level reasoning domain-specific knowledge System-2-capable models MMVU video understanding