MedQ-Bench: Evaluating and Exploring Medical Image Quality Assessment
  Abilities in MLLMs
Medical Image Quality Assessment (IQA) serves as the first-mile safety gate
for clinical AI, yet existing approaches remain constrained by scalar,
score-based metrics and fail to reflect the descriptive, human-like reasoning
process central to expert evaluation. To address this gap, we introduce
MedQ-Bench, a comprehensive benchmark that establishes a perception-reasoning
paradigm for language-based evaluation of medical image quality with
Multi-modal Large Language Models (MLLMs). MedQ-Bench defines two complementary
tasks: (1) MedQ-Perception, which probes low-level perceptual capability via
human-curated questions on fundamental visual attributes; and (2)
MedQ-Reasoning, encompassing both no-reference and comparison reasoning tasks,
aligning model evaluation with human-like reasoning on image quality. The
benchmark spans five imaging modalities and over forty quality attributes,
totaling 2,600 perceptual queries and 708 reasoning assessments, covering
diverse image sources including authentic clinical acquisitions, images with
simulated degradations via physics-based reconstructions, and AI-generated
images. To evaluate reasoning ability, we propose a multi-dimensional judging
protocol that assesses model outputs along four complementary axes. We further
conduct rigorous human-AI alignment validation by comparing LLM-based judgement
with radiologists. Our evaluation of 14 state-of-the-art MLLMs demonstrates
that models exhibit preliminary but unstable perceptual and reasoning skills,
with insufficient accuracy for reliable clinical use. These findings highlight
the need for targeted optimization of MLLMs in medical IQA. We hope that
MedQ-Bench will catalyze further exploration and unlock the untapped potential
of MLLMs for medical image quality evaluation.
68df2d8edf49fb0df1e03bfd
Jiyao Liu
68df2d8edf49fb0df1e03bfe
Jinjie Wei
68df2d8edf49fb0df1e03bff
Wanying Qu
68df2d8edf49fb0df1e03c00
Chenglong Ma
68df2d8edf49fb0df1e03c01
Junzhi Ning
claimed_verified
2025-10-22T15:46:57.366Z
67193c60f89011dc142caf07
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/HC3fjL9V0hhpmjc4kWnGF.png
Junzhi Ning
user
junzhin
68df2d8edf49fb0df1e03c02
Yunheng Li
68df2d8edf49fb0df1e03c03
Ying Chen
68df2d8edf49fb0df1e03c04
Xinzhe Luo
68df2d8edf49fb0df1e03c05
Pengcheng Chen
68df2d8edf49fb0df1e03c06
Xin Gao
68df2d8edf49fb0df1e03c07
Ming Hu
68df2d8edf49fb0df1e03c08
Huihui Xu
68df2d8edf49fb0df1e03c09
Xin Wang
68df2d8edf49fb0df1e03c0a
Shujian Gao
68df2d8edf49fb0df1e03c0b
Dingkang Yang
68df2d8edf49fb0df1e03c0c
Zhongying Deng
68df2d8edf49fb0df1e03c0d
Jin Ye
68df2d8edf49fb0df1e03c0e
Lihao Liu
68df2d8edf49fb0df1e03c0f
Junjun He
68df2d8edf49fb0df1e03c10
Ningsheng Xu
MedQ-Bench: Evaluating and Exploring Medical Image Quality Assessment
  Abilities in MLLMs
Medical Image Quality Assessment (IQA) serves as the first-mile safety gate
for clinical AI, yet existing approaches remain constrained by scalar,
score-based metrics and fail to reflect the descriptive, human-like reasoning
process central to expert evaluation. To address this gap, we introduce
MedQ-Bench, a comprehensive benchmark that establishes a perception-reasoning
paradigm for language-based evaluation of medical image quality with
Multi-modal Large Language Models (MLLMs). MedQ-Bench defines two complementary
tasks: (1) MedQ-Perception, which probes low-level perceptual capability via
human-curated questions on fundamental visual attributes; and (2)
MedQ-Reasoning, encompassing both no-reference and comparison reasoning tasks,
aligning model evaluation with human-like reasoning on image quality. The
benchmark spans five imaging modalities and over forty quality attributes,
totaling 2,600 perceptual queries and 708 reasoning assessments, covering
diverse image sources including authentic clinical acquisitions, images with
simulated degradations via physics-based reconstructions, and AI-generated
images. To evaluate reasoning ability, we propose a multi-dimensional judging
protocol that assesses model outputs along four complementary axes. We further
conduct rigorous human-AI alignment validation by comparing LLM-based judgement
with radiologists. Our evaluation of 14 state-of-the-art MLLMs demonstrates
that models exhibit preliminary but unstable perceptual and reasoning skills,
with insufficient accuracy for reliable clinical use. These findings highlight
the need for targeted optimization of MLLMs in medical IQA. We hope that
MedQ-Bench will catalyze further exploration and unlock the untapped potential
of MLLMs for medical image quality evaluation.
MedQ-Bench introduces a benchmark for language-based evaluation of medical image quality using Multi-modal Large Language Models, focusing on both perceptual and reasoning capabilities.
Multi-modal Large Language Models MedQ-Bench MedQ-Perception MedQ-Reasoning perception-reasoning paradigm human-curated questions visual attributes no-reference reasoning comparison reasoning multi-dimensional judging protocol human-AI alignment validation