MedBLINK: Probing Basic Perception in Multimodal Language Models for
  Medicine
Multimodal language models (MLMs) show promise for clinical decision support
and diagnostic reasoning, raising the prospect of end-to-end automated medical
image interpretation. However, clinicians are highly selective in adopting AI
tools; a model that makes errors on seemingly simple perception tasks such as
determining image orientation or identifying whether a CT scan is
contrast-enhance are unlikely to be adopted for clinical tasks. We introduce
Medblink, a benchmark designed to probe these models for such perceptual
abilities. Medblink spans eight clinically meaningful tasks across multiple
imaging modalities and anatomical regions, totaling 1,429 multiple-choice
questions over 1,605 images. We evaluate 19 state-of-the-art MLMs, including
general purpose (GPT4o, Claude 3.5 Sonnet) and domain specific (Med Flamingo,
LLaVA Med, RadFM) models. While human annotators achieve 96.4% accuracy, the
best-performing model reaches only 65%. These results show that current MLMs
frequently fail at routine perceptual checks, suggesting the need to strengthen
their visual grounding to support clinical adoption. Data is available on our
project page.
6895505748b0ae5ca2710cd3
Mahtab Bigverdi
claimed_verified
2025-08-08T16:20:32.618Z
648380d1f9256e215de1ff88
/avatars/a609c74911ed4f96843debdf2e7568ca.svg
Mahtab Bigverdi
user
MahtabBg
6895505748b0ae5ca2710cd4
Wisdom Ikezogwo
6895505748b0ae5ca2710cd5
Kevin Zhang
6895505748b0ae5ca2710cd6
Hyewon Jeong
6895505748b0ae5ca2710cd7
Mingyu Lu
6895505748b0ae5ca2710cd8
Sungjae Cho
6895505748b0ae5ca2710cd9
Linda Shapiro
6895505748b0ae5ca2710cda
Ranjay Krishna
MedBLINK: Probing Basic Perception in Multimodal Language Models for
  Medicine
Multimodal language models (MLMs) show promise for clinical decision support
and diagnostic reasoning, raising the prospect of end-to-end automated medical
image interpretation. However, clinicians are highly selective in adopting AI
tools; a model that makes errors on seemingly simple perception tasks such as
determining image orientation or identifying whether a CT scan is
contrast-enhance are unlikely to be adopted for clinical tasks. We introduce
Medblink, a benchmark designed to probe these models for such perceptual
abilities. Medblink spans eight clinically meaningful tasks across multiple
imaging modalities and anatomical regions, totaling 1,429 multiple-choice
questions over 1,605 images. We evaluate 19 state-of-the-art MLMs, including
general purpose (GPT4o, Claude 3.5 Sonnet) and domain specific (Med Flamingo,
LLaVA Med, RadFM) models. While human annotators achieve 96.4% accuracy, the
best-performing model reaches only 65%. These results show that current MLMs
frequently fail at routine perceptual checks, suggesting the need to strengthen
their visual grounding to support clinical adoption. Data is available on our
project page.
Medblink benchmark evaluates the perceptual abilities of multimodal language models in clinical image interpretation, revealing significant gaps compared to human performance.
multimodal language models MLMS clinical decision support diagnostic reasoning end-to-end automated medical image interpretation Medblink perceptual abilities clinically meaningful tasks imaging modalities anatomical regions multiple-choice questions general purpose models domain specific models visual grounding