Segment Anything with Multiple Modalities
Robust and accurate segmentation of scenes has become one core functionality
in various visual recognition and navigation tasks. This has inspired the
recent development of Segment Anything Model (SAM), a foundation model for
general mask segmentation. However, SAM is largely tailored for single-modal
RGB images, limiting its applicability to multi-modal data captured with
widely-adopted sensor suites, such as LiDAR plus RGB, depth plus RGB, thermal
plus RGB, etc. We develop MM-SAM, an extension and expansion of SAM that
supports cross-modal and multi-modal processing for robust and enhanced
segmentation with different sensor suites. MM-SAM features two key designs,
namely, unsupervised cross-modal transfer and weakly-supervised multi-modal
fusion, enabling label-efficient and parameter-efficient adaptation toward
various sensor modalities. It addresses three main challenges: 1) adaptation
toward diverse non-RGB sensors for single-modal processing, 2) synergistic
processing of multi-modal data via sensor fusion, and 3) mask-free training for
different downstream tasks. Extensive experiments show that MM-SAM consistently
outperforms SAM by large margins, demonstrating its effectiveness and
robustness across various sensors and data modalities.
66c41719c55655c7158d05ac
Aoran Xiao
66c41719c55655c7158d05ad
Weihao Xuan
claimed_verified
2024-08-27T07:44:15.423Z
65b8909c89eb3dfbe8d26780
https://cdn-avatars.huggingface.co/v1/production/uploads/65b8909c89eb3dfbe8d26780/aJExXsCN9oQOdd9bsQOEO.jpeg
Weihao XUAN
user
weihao1115
66c41719c55655c7158d05ae
Heli Qi
claimed_verified
2024-10-24T07:16:13.148Z
6682a6347ad6fe37b1c5a1d7
/avatars/3dce84bddb82a8647580a5006c0600c0.svg
Heli Qi
user
qiheli1998
66c41719c55655c7158d05af
Yun Xing
claimed_verified
2024-10-22T08:03:30.520Z
65386066bf790526ef358701
/avatars/f02b48f1b05c353e5ab371de45b9cf4c.svg
Xing Yun
user
xing0047
66c41719c55655c7158d05b0
Naoto Yokoya
66c41719c55655c7158d05b1
Shijian Lu
Segment Anything with Multiple Modalities
Robust and accurate segmentation of scenes has become one core functionality
in various visual recognition and navigation tasks. This has inspired the
recent development of Segment Anything Model (SAM), a foundation model for
general mask segmentation. However, SAM is largely tailored for single-modal
RGB images, limiting its applicability to multi-modal data captured with
widely-adopted sensor suites, such as LiDAR plus RGB, depth plus RGB, thermal
plus RGB, etc. We develop MM-SAM, an extension and expansion of SAM that
supports cross-modal and multi-modal processing for robust and enhanced
segmentation with different sensor suites. MM-SAM features two key designs,
namely, unsupervised cross-modal transfer and weakly-supervised multi-modal
fusion, enabling label-efficient and parameter-efficient adaptation toward
various sensor modalities. It addresses three main challenges: 1) adaptation
toward diverse non-RGB sensors for single-modal processing, 2) synergistic
processing of multi-modal data via sensor fusion, and 3) mask-free training for
different downstream tasks. Extensive experiments show that MM-SAM consistently
outperforms SAM by large margins, demonstrating its effectiveness and
robustness across various sensors and data modalities.
MM-SAM extends SAM to support multi-modal and cross-modal processing, enhancing segmentation accuracy across various sensor modalities through unsupervised cross-modal transfer and weakly-supervised multi-modal fusion.
Segment Anything Model SAM MM-SAM cross-modal transfer weakly-supervised multi-modal fusion multi-modal data sensor fusion mask-free training single-modal processing sensor modalities