M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for
  Evaluating Foundation Models
Existing benchmarks for evaluating foundation models mainly focus on
single-document, text-only tasks. However, they often fail to fully capture the
complexity of research workflows, which typically involve interpreting
non-textual data and gathering information across multiple documents. To
address this gap, we introduce M3SciQA, a multi-modal, multi-document
scientific question answering benchmark designed for a more comprehensive
evaluation of foundation models. M3SciQA consists of 1,452 expert-annotated
questions spanning 70 natural language processing paper clusters, where each
cluster represents a primary paper along with all its cited documents,
mirroring the workflow of comprehending a single paper by requiring multi-modal
and multi-document data. With M3SciQA, we conduct a comprehensive evaluation of
18 foundation models. Our results indicate that current foundation models still
significantly underperform compared to human experts in multi-modal information
retrieval and in reasoning across multiple scientific documents. Additionally,
we explore the implications of these findings for the future advancement of
applying foundation models in multi-modal scientific literature analysis.
672cd87096a9d93d169922a8
Chuhan Li
claimed_verified
2024-11-15T09:33:09.809Z
65415f1d5168c4f3487a2103
https://cdn-avatars.huggingface.co/v1/production/uploads/65415f1d5168c4f3487a2103/qJuzDpOGSDL4E1-L2lGWW.jpeg
Chuhan Li
user
ChuhanLi
672cd87096a9d93d169922a9
Ziyao Shangguan
claimed_verified
2024-11-15T09:33:04.816Z
65dea56779f827c045b1df96
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Jtx4JcM5CNgoVyzxTC93S.jpeg
Ziyao Shangguan
user
ziyaosg
672cd87096a9d93d169922aa
Yilun Zhao
claimed_verified
2024-12-03T11:16:38.117Z
62f662bcc58915315c4eccea
https://cdn-avatars.huggingface.co/v1/production/uploads/62f662bcc58915315c4eccea/zOAQLONfMP88zr70sxHK-.jpeg
Yilun Zhao
user
yilunzhao
672cd87096a9d93d169922ab
Deyuan Li
672cd87096a9d93d169922ac
Yixin Liu
672cd87096a9d93d169922ad
Arman Cohan
M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for
  Evaluating Foundation Models
Existing benchmarks for evaluating foundation models mainly focus on
single-document, text-only tasks. However, they often fail to fully capture the
complexity of research workflows, which typically involve interpreting
non-textual data and gathering information across multiple documents. To
address this gap, we introduce M3SciQA, a multi-modal, multi-document
scientific question answering benchmark designed for a more comprehensive
evaluation of foundation models. M3SciQA consists of 1,452 expert-annotated
questions spanning 70 natural language processing paper clusters, where each
cluster represents a primary paper along with all its cited documents,
mirroring the workflow of comprehending a single paper by requiring multi-modal
and multi-document data. With M3SciQA, we conduct a comprehensive evaluation of
18 foundation models. Our results indicate that current foundation models still
significantly underperform compared to human experts in multi-modal information
retrieval and in reasoning across multiple scientific documents. Additionally,
we explore the implications of these findings for the future advancement of
applying foundation models in multi-modal scientific literature analysis.
M3SciQA evaluates foundation models on multi-modal, multi-document scientific question answering, showing significant performance gaps compared to human experts.
multi-modal multi-document scientific question answering foundation models information retrieval natural language processing