PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal
  Inconsistencies
Large Multimodal Models (LMMs) are increasingly applied to scientific
research, yet it remains unclear whether they can reliably understand and
reason over the multimodal complexity of papers. A central challenge lies in
detecting and resolving inconsistencies across text, figures, tables, and
equations, issues that are often subtle, domain-specific, and ultimately
undermine clarity, reproducibility, and trust. Existing benchmarks overlook
this issue, either isolating single modalities or relying on synthetic errors
that fail to capture real-world complexity. We introduce PRISMM-Bench
(Peer-Review-sourced Inconsistency Set for Multimodal Models), the first
benchmark grounded in real reviewer-flagged inconsistencies in scientific
papers. Through a multi-stage pipeline of review mining, LLM-assisted filtering
and human verification, we curate 262 inconsistencies from 242 papers. Based on
this set, we design three tasks, namely inconsistency identification, remedy
and pair matching, which assess a model's capacity to detect, correct, and
reason over inconsistencies across different modalities. Furthermore, to
address the notorious problem of choice-only shortcuts in multiple-choice
evaluation, where models exploit answer patterns without truly understanding
the question, we further introduce structured JSON-based answer representations
that minimize linguistic biases by reducing reliance on superficial stylistic
cues. We benchmark 21 leading LMMs, including large open-weight models
(GLM-4.5V 106B, InternVL3 78B) and proprietary models (Gemini 2.5 Pro, GPT-5
with high reasoning). Results reveal strikingly low performance (26.1-54.2%),
underscoring the challenge of multimodal scientific reasoning and motivating
progress towards trustworthy scientific assistants.
68f72b4124c4489363111a20
Lukas Selch
68f72b4124c4489363111a21
Yufang Hou
68f72b4124c4489363111a22
M. Jehanzeb Mirza
68f72b4124c4489363111a23
Sivan Doveh
68f72b4124c4489363111a24
James Glass
68f72b4124c4489363111a25
Rogerio Feris
68f72b4124c4489363111a26
Wei Lin
PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal
  Inconsistencies
Large Multimodal Models (LMMs) are increasingly applied to scientific
research, yet it remains unclear whether they can reliably understand and
reason over the multimodal complexity of papers. A central challenge lies in
detecting and resolving inconsistencies across text, figures, tables, and
equations, issues that are often subtle, domain-specific, and ultimately
undermine clarity, reproducibility, and trust. Existing benchmarks overlook
this issue, either isolating single modalities or relying on synthetic errors
that fail to capture real-world complexity. We introduce PRISMM-Bench
(Peer-Review-sourced Inconsistency Set for Multimodal Models), the first
benchmark grounded in real reviewer-flagged inconsistencies in scientific
papers. Through a multi-stage pipeline of review mining, LLM-assisted filtering
and human verification, we curate 262 inconsistencies from 242 papers. Based on
this set, we design three tasks, namely inconsistency identification, remedy
and pair matching, which assess a model's capacity to detect, correct, and
reason over inconsistencies across different modalities. Furthermore, to
address the notorious problem of choice-only shortcuts in multiple-choice
evaluation, where models exploit answer patterns without truly understanding
the question, we further introduce structured JSON-based answer representations
that minimize linguistic biases by reducing reliance on superficial stylistic
cues. We benchmark 21 leading LMMs, including large open-weight models
(GLM-4.5V 106B, InternVL3 78B) and proprietary models (Gemini 2.5 Pro, GPT-5
with high reasoning). Results reveal strikingly low performance (26.1-54.2%),
underscoring the challenge of multimodal scientific reasoning and motivating
progress towards trustworthy scientific assistants.
PRISMM-Bench evaluates the ability of large multimodal models to detect, correct, and reason over inconsistencies in scientific papers, revealing significant challenges in multimodal scientific reasoning.
Large Multimodal Models PRISMM-Bench inconsistency identification remedy pair matching review mining LLM-assisted filtering human verification structured JSON-based answer representations multimodal scientific reasoning trustworthy scientific assistants