Mamba as a Bridge: Where Vision Foundation Models Meet Vision Language
  Models for Domain-Generalized Semantic Segmentation
Vision Foundation Models (VFMs) and Vision-Language Models (VLMs) have gained
traction in Domain Generalized Semantic Segmentation (DGSS) due to their strong
generalization capabilities. However, existing DGSS methods often rely
exclusively on either VFMs or VLMs, overlooking their complementary strengths.
VFMs (e.g., DINOv2) excel at capturing fine-grained features, while VLMs (e.g.,
CLIP) provide robust text alignment but struggle with coarse granularity.
Despite their complementary strengths, effectively integrating VFMs and VLMs
with attention mechanisms is challenging, as the increased patch tokens
complicate long-sequence modeling. To address this, we propose MFuser, a novel
Mamba-based fusion framework that efficiently combines the strengths of VFMs
and VLMs while maintaining linear scalability in sequence length. MFuser
consists of two key components: MVFuser, which acts as a co-adapter to jointly
fine-tune the two models by capturing both sequential and spatial dynamics; and
MTEnhancer, a hybrid attention-Mamba module that refines text embeddings by
incorporating image priors. Our approach achieves precise feature locality and
strong text alignment without incurring significant computational overhead.
Extensive experiments demonstrate that MFuser significantly outperforms
state-of-the-art DGSS methods, achieving 68.20 mIoU on synthetic-to-real and
71.87 mIoU on real-to-real benchmarks. The code is available at
https://github.com/devinxzhang/MFuser.
67f3f25019592b36b6d8b30c
Xin Zhang
claimed_verified
2025-04-08T06:52:23.735Z
6436290e76dbfd731bcf1f55
/avatars/63af84ca382d799c4a000594c994d4bb.svg
Xin Zhang
user
XinNUS
67f3f25019592b36b6d8b30d
Robby T. Tan
admin_assigned
2025-04-08T10:39:06.222Z
67c88df8eeb202fffea14efd
/avatars/915f516389c99f03495b98a41fca704b.svg
robbytan
user
robbytan
Mamba as a Bridge: Where Vision Foundation Models Meet Vision Language
  Models for Domain-Generalized Semantic Segmentation
Vision Foundation Models (VFMs) and Vision-Language Models (VLMs) have gained
traction in Domain Generalized Semantic Segmentation (DGSS) due to their strong
generalization capabilities. However, existing DGSS methods often rely
exclusively on either VFMs or VLMs, overlooking their complementary strengths.
VFMs (e.g., DINOv2) excel at capturing fine-grained features, while VLMs (e.g.,
CLIP) provide robust text alignment but struggle with coarse granularity.
Despite their complementary strengths, effectively integrating VFMs and VLMs
with attention mechanisms is challenging, as the increased patch tokens
complicate long-sequence modeling. To address this, we propose MFuser, a novel
Mamba-based fusion framework that efficiently combines the strengths of VFMs
and VLMs while maintaining linear scalability in sequence length. MFuser
consists of two key components: MVFuser, which acts as a co-adapter to jointly
fine-tune the two models by capturing both sequential and spatial dynamics; and
MTEnhancer, a hybrid attention-Mamba module that refines text embeddings by
incorporating image priors. Our approach achieves precise feature locality and
strong text alignment without incurring significant computational overhead.
Extensive experiments demonstrate that MFuser significantly outperforms
state-of-the-art DGSS methods, achieving 68.20 mIoU on synthetic-to-real and
71.87 mIoU on real-to-real benchmarks. The code is available at
https://github.com/devinxzhang/MFuser.
MFuser combines Vision Foundation Models and Vision-Language Models using Mamba-based fusion for enhanced Domain Generalized Semantic Segmentation, achieving high performance with efficient scalability.
Vision Foundation Models Vision-Language Models Domain Generalized Semantic Segmentation DINOv2 CLIP patch tokens long-sequence modeling Mamba-based fusion MFuser MVFuser MTEnhancer hybrid attention-Mamba feature locality text alignment mIoU