MedSAMix: A Training-Free Model Merging Approach for Medical Image
  Segmentation
Universal medical image segmentation models have emerged as a promising
paradigm due to their strong generalizability across diverse tasks, showing
great potential for a wide range of clinical applications. This potential has
been partly driven by the success of general-purpose vision models such as the
Segment Anything Model (SAM), which has inspired the development of various
fine-tuned variants for medical segmentation tasks. However, fine-tuned
variants like MedSAM are trained on comparatively limited medical imaging data
that often suffers from heterogeneity, scarce annotations, and distributional
shifts. These challenges limit their ability to generalize across a wide range
of medical segmentation tasks. In this regard, we propose MedSAMix, a
training-free model merging method that integrates the strengths of both
generalist models (e.g., SAM) and specialist models (e.g., MedSAM) for medical
image segmentation. In contrast to traditional model merging approaches that
rely on manual configuration and often result in suboptimal outcomes, we
propose a zero-order optimization method to automatically discover optimal
layer-wise merging solutions. Furthermore, for clinical applications, we
develop two regimes to meet the demand of domain-specificity and
generalizability in different scenarios by single-task optimization and
multi-objective optimization respectively. Extensive evaluations on 25 medical
segmentation tasks demonstrate that MedSAMix effectively mitigates model bias
and consistently improves performance in both domain-specific accuracy and
generalization, achieving improvements of 6.67% on specialized tasks and 4.37%
on multi-task evaluations.
68a2e436a4caabb4320e64ee
Yanwu Yang
claimed_verified
2025-08-20T08:53:11.002Z
65bf56b90a0c57943f0540aa
/avatars/7f92f81cd9084f615dcf27943d377722.svg
Yanwu
user
podismine
68a2e436a4caabb4320e64ef
Guinan Su
claimed_verified
2025-08-20T08:53:07.865Z
630ed6cfc9af0163b96a71aa
/avatars/bf984bb1c2f68336bcf042c6fc7e0d9d.svg
Guinan-Su
user
guinansu
68a2e436a4caabb4320e64f0
Jiesi Hu
68a2e436a4caabb4320e64f1
Francesco Sammarco
68a2e436a4caabb4320e64f2
Jonas Geiping
68a2e436a4caabb4320e64f3
Thomas Wolfers
MedSAMix: A Training-Free Model Merging Approach for Medical Image
  Segmentation
Universal medical image segmentation models have emerged as a promising
paradigm due to their strong generalizability across diverse tasks, showing
great potential for a wide range of clinical applications. This potential has
been partly driven by the success of general-purpose vision models such as the
Segment Anything Model (SAM), which has inspired the development of various
fine-tuned variants for medical segmentation tasks. However, fine-tuned
variants like MedSAM are trained on comparatively limited medical imaging data
that often suffers from heterogeneity, scarce annotations, and distributional
shifts. These challenges limit their ability to generalize across a wide range
of medical segmentation tasks. In this regard, we propose MedSAMix, a
training-free model merging method that integrates the strengths of both
generalist models (e.g., SAM) and specialist models (e.g., MedSAM) for medical
image segmentation. In contrast to traditional model merging approaches that
rely on manual configuration and often result in suboptimal outcomes, we
propose a zero-order optimization method to automatically discover optimal
layer-wise merging solutions. Furthermore, for clinical applications, we
develop two regimes to meet the demand of domain-specificity and
generalizability in different scenarios by single-task optimization and
multi-objective optimization respectively. Extensive evaluations on 25 medical
segmentation tasks demonstrate that MedSAMix effectively mitigates model bias
and consistently improves performance in both domain-specific accuracy and
generalization, achieving improvements of 6.67% on specialized tasks and 4.37%
on multi-task evaluations.
MedSAMix, a training-free model merging method, combines generalist and specialist models to improve medical image segmentation performance across diverse tasks.
Segment Anything Model (SAM) MedSAM model merging zero-order optimization layer-wise merging domain-specificity generalizability single-task optimization multi-objective optimization medical segmentation tasks