SciVer: Evaluating Foundation Models for Multimodal Scientific Claim
  Verification
We introduce SciVer, the first benchmark specifically designed to evaluate
the ability of foundation models to verify claims within a multimodal
scientific context. SciVer consists of 3,000 expert-annotated examples over
1,113 scientific papers, covering four subsets, each representing a common
reasoning type in multimodal scientific claim verification. To enable
fine-grained evaluation, each example includes expert-annotated supporting
evidence. We assess the performance of 21 state-of-the-art multimodal
foundation models, including o4-mini, Gemini-2.5-Flash, Llama-3.2-Vision, and
Qwen2.5-VL. Our experiment reveals a substantial performance gap between these
models and human experts on SciVer. Through an in-depth analysis of
retrieval-augmented generation (RAG), and human-conducted error evaluations, we
identify critical limitations in current open-source models, offering key
insights to advance models' comprehension and reasoning in multimodal
scientific literature tasks.
68537ca999bf39f9665c799a
Chengye Wang
68537ca999bf39f9665c799b
Yifei Shen
68537ca999bf39f9665c799c
Zexi Kuang
68537ca999bf39f9665c799d
Arman Cohan
68537ca999bf39f9665c799e
Yilun Zhao
claimed_verified
2025-06-19T09:10:23.732Z
62f662bcc58915315c4eccea
https://cdn-avatars.huggingface.co/v1/production/uploads/62f662bcc58915315c4eccea/zOAQLONfMP88zr70sxHK-.jpeg
Yilun Zhao
user
yilunzhao
SciVer: Evaluating Foundation Models for Multimodal Scientific Claim
  Verification
We introduce SciVer, the first benchmark specifically designed to evaluate
the ability of foundation models to verify claims within a multimodal
scientific context. SciVer consists of 3,000 expert-annotated examples over
1,113 scientific papers, covering four subsets, each representing a common
reasoning type in multimodal scientific claim verification. To enable
fine-grained evaluation, each example includes expert-annotated supporting
evidence. We assess the performance of 21 state-of-the-art multimodal
foundation models, including o4-mini, Gemini-2.5-Flash, Llama-3.2-Vision, and
Qwen2.5-VL. Our experiment reveals a substantial performance gap between these
models and human experts on SciVer. Through an in-depth analysis of
retrieval-augmented generation (RAG), and human-conducted error evaluations, we
identify critical limitations in current open-source models, offering key
insights to advance models' comprehension and reasoning in multimodal
scientific literature tasks.
A benchmark named SciVer evaluates multimodal foundation models' claim verification capabilities within scientific contexts, revealing performance gaps and limitations in current models.
retrieval-augmented generation (RAG)