SciVer: Evaluating Foundation Models for Multimodal Scientific Claim
  Verification
We introduce SciVer, the first benchmark specifically designed to evaluate
the ability of foundation models to verify claims within a multimodal
scientific context. SciVer consists of 3,000 expert-annotated examples over
1,113 scientific papers, covering four subsets, each representing a common
reasoning type in multimodal scientific claim verification. To enable
fine-grained evaluation, each example includes expert-annotated supporting
evidence. We assess the performance of 21 state-of-the-art multimodal
foundation models, including o4-mini, Gemini-2.5-Flash, Llama-3.2-Vision, and
Qwen2.5-VL. Our experiment reveals a substantial performance gap between these
models and human experts on SciVer. Through an in-depth analysis of
retrieval-augmented generation (RAG), and human-conducted error evaluations, we
identify critical limitations in current open-source models, offering key
insights to advance models' comprehension and reasoning in multimodal
scientific literature tasks.
68537ca999bf39f9665c799a
Chengye Wang
68537ca999bf39f9665c799b
Yifei Shen
claimed_verified
2025-09-29T06:06:34.781Z
649aa367c6cf3cc95bc1b7f6
/avatars/4bf5446c261eab08fc06caebf4c5779a.svg
Yifei Shen
user
yshenaw
68537ca999bf39f9665c799c
Zexi Kuang
68537ca999bf39f9665c799d
Arman Cohan
68537ca999bf39f9665c799e
Yilun Zhao
claimed_verified
2025-06-19T09:10:23.732Z
62f662bcc58915315c4eccea
https://cdn-avatars.huggingface.co/v1/production/uploads/62f662bcc58915315c4eccea/zOAQLONfMP88zr70sxHK-.jpeg
Yilun Zhao
user
yilunzhao
SciVer: Evaluating Foundation Models for Multimodal Scientific Claim
  Verification
We introduce SciVer, the first benchmark specifically designed to evaluate
the ability of foundation models to verify claims within a multimodal
scientific context. SciVer consists of 3,000 expert-annotated examples over
1,113 scientific papers, covering four subsets, each representing a common
reasoning type in multimodal scientific claim verification. To enable
fine-grained evaluation, each example includes expert-annotated supporting
evidence. We assess the performance of 21 state-of-the-art multimodal
foundation models, including o4-mini, Gemini-2.5-Flash, Llama-3.2-Vision, and
Qwen2.5-VL. Our experiment reveals a substantial performance gap between these
models and human experts on SciVer. Through an in-depth analysis of
retrieval-augmented generation (RAG), and human-conducted error evaluations, we
identify critical limitations in current open-source models, offering key
insights to advance models' comprehension and reasoning in multimodal
scientific literature tasks.
A benchmark named SciVer evaluates multimodal foundation models' claim verification capabilities within scientific contexts, revealing performance gaps and limitations in current models.
retrieval-augmented generation (RAG)