Enhancing Abnormality Grounding for Vision Language Models with
  Knowledge Descriptions
Visual Language Models (VLMs) have demonstrated impressive capabilities in
visual grounding tasks. However, their effectiveness in the medical domain,
particularly for abnormality detection and localization within medical images,
remains underexplored. A major challenge is the complex and abstract nature of
medical terminology, which makes it difficult to directly associate
pathological anomaly terms with their corresponding visual features. In this
work, we introduce a novel approach to enhance VLM performance in medical
abnormality detection and localization by leveraging decomposed medical
knowledge. Instead of directly prompting models to recognize specific
abnormalities, we focus on breaking down medical concepts into fundamental
attributes and common visual patterns. This strategy promotes a stronger
alignment between textual descriptions and visual features, improving both the
recognition and localization of abnormalities in medical images.We evaluate our
method on the 0.23B Florence-2 base model and demonstrate that it achieves
comparable performance in abnormality grounding to significantly larger 7B
LLaVA-based medical VLMs, despite being trained on only 1.5% of the data used
for such models. Experimental results also demonstrate the effectiveness of our
approach in both known and previously unseen abnormalities, suggesting its
strong generalization capabilities.
67c94e5f8c4ef8be73583f4b
Jun Li
admin_assigned
2025-04-09T15:59:20.528Z
6526523fa754153bd4da45f3
/avatars/848b234fcd59d9885ddb673ffea20a63.svg
RioJune
user
RioJune
67c94e5f8c4ef8be73583f4c
Che Liu
admin_assigned
2025-03-06T10:03:06.882Z
631b9ff5824f2502e3557c7e
/avatars/076043c9dba07644a570692563ef8114.svg
liu
user
che111
67c94e5f8c4ef8be73583f4d
Wenjia Bai
67c94e5f8c4ef8be73583f4e
Rossella Arcucci
67c94e5f8c4ef8be73583f4f
Cosmin I. Bercea
67c94e5f8c4ef8be73583f50
Julia A. Schnabel
Enhancing Abnormality Grounding for Vision Language Models with
  Knowledge Descriptions
Visual Language Models (VLMs) have demonstrated impressive capabilities in
visual grounding tasks. However, their effectiveness in the medical domain,
particularly for abnormality detection and localization within medical images,
remains underexplored. A major challenge is the complex and abstract nature of
medical terminology, which makes it difficult to directly associate
pathological anomaly terms with their corresponding visual features. In this
work, we introduce a novel approach to enhance VLM performance in medical
abnormality detection and localization by leveraging decomposed medical
knowledge. Instead of directly prompting models to recognize specific
abnormalities, we focus on breaking down medical concepts into fundamental
attributes and common visual patterns. This strategy promotes a stronger
alignment between textual descriptions and visual features, improving both the
recognition and localization of abnormalities in medical images.We evaluate our
method on the 0.23B Florence-2 base model and demonstrate that it achieves
comparable performance in abnormality grounding to significantly larger 7B
LLaVA-based medical VLMs, despite being trained on only 1.5% of the data used
for such models. Experimental results also demonstrate the effectiveness of our
approach in both known and previously unseen abnormalities, suggesting its
strong generalization capabilities.
A new approach decomposes medical knowledge into fundamental attributes and visual patterns to enhance visual language models in medical abnormality detection and localization, achieving comparable performance to larger models with less data.
Visual Language Models VLMs abnormality detection localization medical images decomposed medical knowledge textual descriptions visual features Florence-2 base model LLaVA-based models