On the Compositional Generalization of Multimodal LLMs for Medical
  Imaging
Multimodal large language models (MLLMs) hold significant potential in the
medical field, but their capabilities are often limited by insufficient data in
certain medical domains, highlighting the need for understanding what kinds of
images can be used by MLLMs for generalization. Current research suggests that
multi-task training outperforms single-task as different tasks can benefit each
other, but they often overlook the internal relationships within these tasks,
providing limited guidance on selecting datasets to enhance specific tasks. To
analyze this phenomenon, we attempted to employ compositional generalization
(CG)-the ability of models to understand novel combinations by recombining
learned elements-as a guiding framework. Since medical images can be precisely
defined by Modality, Anatomical area, and Task, naturally providing an
environment for exploring CG. Therefore, we assembled 106 medical datasets to
create Med-MAT for comprehensive experiments. The experiments confirmed that
MLLMs can use CG to understand unseen medical images and identified CG as one
of the main drivers of the generalization observed in multi-task training.
Additionally, further studies demonstrated that CG effectively supports
datasets with limited data and delivers consistent performance across different
backbones, highlighting its versatility and broad applicability. Med-MAT is
publicly available at https://github.com/FreedomIntelligence/Med-MAT.
67735f9f9cc5d33bf6af3cef
Zhenyang Cai
claimed_verified
2025-01-01T20:13:04.716Z
64f1a34f2c5c8b767916447e
https://cdn-avatars.huggingface.co/v1/production/uploads/64f1a34f2c5c8b767916447e/uak2CsMAnxW8q4dwyAOBN.jpeg
Zhenyang Cai
user
Eric3200
67735f9f9cc5d33bf6af3cf0
Junying Chen
67735f9f9cc5d33bf6af3cf1
Rongsheng Wang
claimed_verified
2025-01-01T20:13:02.316Z
63ca949b04c979828315389d
https://cdn-avatars.huggingface.co/v1/production/uploads/63ca949b04c979828315389d/HS5xWNAYjjHeyAAwWJ11l.jpeg
wangrongsheng
user
wangrongsheng
67735f9f9cc5d33bf6af3cf2
Weihong Wang
67735f9f9cc5d33bf6af3cf3
Yonglin Deng
67735f9f9cc5d33bf6af3cf4
Dingjie Song
claimed_verified
2025-01-01T20:12:59.736Z
619f01b8cc04eadf54fa5d5d
/avatars/928f3d1a6146e2e1ae4860445d929d5c.svg
Song Dingjie
user
songdj
67735f9f9cc5d33bf6af3cf5
Yize Chen
67735f9f9cc5d33bf6af3cf6
Zixu Zhang
67735f9f9cc5d33bf6af3cf7
Benyou Wang
On the Compositional Generalization of Multimodal LLMs for Medical
  Imaging
Multimodal large language models (MLLMs) hold significant potential in the
medical field, but their capabilities are often limited by insufficient data in
certain medical domains, highlighting the need for understanding what kinds of
images can be used by MLLMs for generalization. Current research suggests that
multi-task training outperforms single-task as different tasks can benefit each
other, but they often overlook the internal relationships within these tasks,
providing limited guidance on selecting datasets to enhance specific tasks. To
analyze this phenomenon, we attempted to employ compositional generalization
(CG)-the ability of models to understand novel combinations by recombining
learned elements-as a guiding framework. Since medical images can be precisely
defined by Modality, Anatomical area, and Task, naturally providing an
environment for exploring CG. Therefore, we assembled 106 medical datasets to
create Med-MAT for comprehensive experiments. The experiments confirmed that
MLLMs can use CG to understand unseen medical images and identified CG as one
of the main drivers of the generalization observed in multi-task training.
Additionally, further studies demonstrated that CG effectively supports
datasets with limited data and delivers consistent performance across different
backbones, highlighting its versatility and broad applicability. Med-MAT is
publicly available at https://github.com/FreedomIntelligence/Med-MAT.
Compositional generalization aids multimodal large language models in understanding and generalizing to unseen medical images across various domains and tasks.
multimodal large language models MLLMs medical field multi-task training single-task training compositional generalization CG Med-MAT medical datasets