Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes
  Interactively
The CLIP and Segment Anything Model (SAM) are remarkable vision foundation
models (VFMs). SAM excels in segmentation tasks across diverse domains, while
CLIP is renowned for its zero-shot recognition capabilities. This paper
presents an in-depth exploration of integrating these two models into a unified
framework. Specifically, we introduce the Open-Vocabulary SAM, a SAM-inspired
model designed for simultaneous interactive segmentation and recognition,
leveraging two unique knowledge transfer modules: SAM2CLIP and CLIP2SAM. The
former adapts SAM's knowledge into the CLIP via distillation and learnable
transformer adapters, while the latter transfers CLIP knowledge into SAM,
enhancing its recognition capabilities. Extensive experiments on various
datasets and detectors show the effectiveness of Open-Vocabulary SAM in both
segmentation and recognition tasks, significantly outperforming the naive
baselines of simply combining SAM and CLIP. Furthermore, aided with image
classification data training, our method can segment and recognize
approximately 22,000 classes.
659b62626da3461e2849234a
Haobo Yuan
admin_assigned
2024-01-08T11:45:19.748Z
6391e41f2e73987364e6bcb2
/avatars/d09a9ee329bb8c3a9e2929d67d24e97d.svg
Haobo Yuan
user
HarborYuan
659b62626da3461e2849234b
Xiangtai Li
admin_assigned
2024-01-08T11:45:43.664Z
63958b4414513eaf9029ebf1
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U1g5H071pWRswGAG9UTpo.png
Xiangtai Li
user
LXT
659b62626da3461e2849234c
Chong Zhou
claimed_verified
2024-01-08T08:47:30.592Z
642bfdbbc885078517154abb
https://cdn-avatars.huggingface.co/v1/production/uploads/642bfdbbc885078517154abb/lLmc8_3VnTyk4qOvl-omI.jpeg
Chong Zhou
user
chongzhou
659b62626da3461e2849234d
Yining Li
admin_assigned
2024-01-08T11:46:10.233Z
649988726677f66c2b486392
/avatars/b649a77370660e129726e29504daba34.svg
Yining Li
user
ly015
659b62626da3461e2849234e
Kai Chen
659b62626da3461e2849234f
Chen Change Loy
Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes
  Interactively
The CLIP and Segment Anything Model (SAM) are remarkable vision foundation
models (VFMs). SAM excels in segmentation tasks across diverse domains, while
CLIP is renowned for its zero-shot recognition capabilities. This paper
presents an in-depth exploration of integrating these two models into a unified
framework. Specifically, we introduce the Open-Vocabulary SAM, a SAM-inspired
model designed for simultaneous interactive segmentation and recognition,
leveraging two unique knowledge transfer modules: SAM2CLIP and CLIP2SAM. The
former adapts SAM's knowledge into the CLIP via distillation and learnable
transformer adapters, while the latter transfers CLIP knowledge into SAM,
enhancing its recognition capabilities. Extensive experiments on various
datasets and detectors show the effectiveness of Open-Vocabulary SAM in both
segmentation and recognition tasks, significantly outperforming the naive
baselines of simply combining SAM and CLIP. Furthermore, aided with image
classification data training, our method can segment and recognize
approximately 22,000 classes.
A unified framework combining SAM and CLIP through knowledge transfer improves performance in both segmentation and recognition tasks.
CLIP Segment Anything Model (SAM) vision foundation models (VFMs) interactive segmentation open-vocabulary SAM SAM2CLIP CLIP2SAM knowledge transfer transformer adapters image classification