ExpVid: A Benchmark for Experiment Video Understanding & Reasoning
Multimodal Large Language Models (MLLMs) hold promise for accelerating
scientific discovery by interpreting complex experimental procedures. However,
their true capabilities are poorly understood, as existing benchmarks neglect
the fine-grained and long-horizon nature of authentic laboratory work,
especially in wet-lab settings. To bridge this gap, we introduce ExpVid, the
first benchmark designed to systematically evaluate MLLMs on scientific
experiment videos. Curated from peer-reviewed video publications, ExpVid
features a new three-level task hierarchy that mirrors the scientific process:
(1) Fine-grained Perception of tools, materials, and actions; (2) Procedural
Understanding of step order and completeness; and (3) Scientific Reasoning that
connects the full experiment to its published conclusions. Our vision-centric
annotation pipeline, combining automated generation with multi-disciplinary
expert validation, ensures that tasks require visual grounding. We evaluate 19
leading MLLMs on ExpVid and find that while they excel at coarse-grained
recognition, they struggle with disambiguating fine details, tracking state
changes over time, and linking experimental procedures to scientific outcomes.
Our results reveal a notable performance gap between proprietary and
open-source models, particularly in high-order reasoning. ExpVid not only
provides a diagnostic tool but also charts a roadmap for developing MLLMs
capable of becoming trustworthy partners in scientific experimentation.
68ee35b69b77b5223f666173
Yicheng Xu
claimed_verified
2025-10-14T14:24:10.982Z
682c163fa17480053339f270
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/UQZW2bM9UEtlVHDUq0yel.png
Yicheng Xu
user
linghan199
68ee35b69b77b5223f666174
Yue Wu
68ee35b69b77b5223f666175
Jiashuo Yu
68ee35b69b77b5223f666176
Ziang Yan
68ee35b69b77b5223f666177
Tianxiang Jiang
68ee35b69b77b5223f666178
Yinan He
68ee35b69b77b5223f666179
Qingsong Zhao
68ee35b69b77b5223f66617a
Kai Chen
68ee35b69b77b5223f66617b
Yu Qiao
68ee35b69b77b5223f66617c
Limin Wang
68ee35b69b77b5223f66617d
Manabu Okumura
68ee35b69b77b5223f66617e
Yi Wang
ExpVid: A Benchmark for Experiment Video Understanding & Reasoning
Multimodal Large Language Models (MLLMs) hold promise for accelerating
scientific discovery by interpreting complex experimental procedures. However,
their true capabilities are poorly understood, as existing benchmarks neglect
the fine-grained and long-horizon nature of authentic laboratory work,
especially in wet-lab settings. To bridge this gap, we introduce ExpVid, the
first benchmark designed to systematically evaluate MLLMs on scientific
experiment videos. Curated from peer-reviewed video publications, ExpVid
features a new three-level task hierarchy that mirrors the scientific process:
(1) Fine-grained Perception of tools, materials, and actions; (2) Procedural
Understanding of step order and completeness; and (3) Scientific Reasoning that
connects the full experiment to its published conclusions. Our vision-centric
annotation pipeline, combining automated generation with multi-disciplinary
expert validation, ensures that tasks require visual grounding. We evaluate 19
leading MLLMs on ExpVid and find that while they excel at coarse-grained
recognition, they struggle with disambiguating fine details, tracking state
changes over time, and linking experimental procedures to scientific outcomes.
Our results reveal a notable performance gap between proprietary and
open-source models, particularly in high-order reasoning. ExpVid not only
provides a diagnostic tool but also charts a roadmap for developing MLLMs
capable of becoming trustworthy partners in scientific experimentation.
ExpVid, a new benchmark for evaluating multimodal large language models on scientific experiment videos, highlights gaps in fine-grained perception, procedural understanding, and scientific reasoning.
Multimodal Large Language Models MLLMs ExpVid benchmark scientific experiment videos three-level task hierarchy fine-grained perception procedural understanding scientific reasoning vision-centric annotation pipeline automated generation multi-disciplinary expert validation coarse-grained recognition fine details state changes high-order reasoning proprietary models open-source models