EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything
  Model
Segment Anything Model (SAM) has attracted widespread attention for its
superior interactive segmentation capabilities with visual prompts while
lacking further exploration of text prompts. In this paper, we empirically
investigate what text prompt encoders (e.g., CLIP or LLM) are good for adapting
SAM for referring expression segmentation and introduce the Early
Vision-language Fusion-based SAM (EVF-SAM). EVF-SAM is a simple yet effective
referring segmentation method which exploits multimodal prompts (i.e., image
and text) and comprises a pre-trained vision-language model to generate
referring prompts and a SAM model for segmentation. Surprisingly, we observe
that: (1) multimodal prompts and (2) vision-language models with early fusion
(e.g., BEIT-3) are beneficial for prompting SAM for accurate referring
segmentation. Our experiments show that the proposed EVF-SAM based on BEIT-3
can obtain state-of-the-art performance on RefCOCO/+/g for referring expression
segmentation and demonstrate the superiority of prompting SAM with early
vision-language fusion. In addition, the proposed EVF-SAM with 1.32B parameters
achieves remarkably higher performance while reducing nearly 82% of parameters
compared to previous SAM methods based on large multimodal models.
66824779d892ee6886af5da2
Yuxuan Zhang
claimed_verified
2024-08-08T07:42:30.084Z
656d3ecdc497edf0a7b00a57
/avatars/7879257bf247203a6dcf099b4076aeb5.svg
YxZhang
user
YxZhang
66824779d892ee6886af5da3
Tianheng Cheng
claimed_verified
2024-07-01T07:46:29.538Z
646b3db131968a60a01e4cf5
https://cdn-avatars.huggingface.co/v1/production/uploads/646b3db131968a60a01e4cf5/DhfdqUYQaD1Qa8Svw996J.jpeg
Tianheng Cheng
user
wondervictor
66824779d892ee6886af5da4
Rui Hu
admin_assigned
2024-07-01T09:43:04.536Z
655bcabecfe086853e2aa22b
/avatars/7fd2e7a5ccbbf25643e349682d2f655f.svg
Rui Hu
user
RuiHu
66824779d892ee6886af5da5
ei Liu
66824779d892ee6886af5da6
Heng Liu
66824779d892ee6886af5da7
Longjin Ran
66824779d892ee6886af5da8
Xiaoxin Chen
admin_assigned
2024-07-01T09:42:39.544Z
65389a669c474315d7425f96
/avatars/2fa3828ca489cfe1948129a0eccf264f.svg
chenxiaoxin
user
steelozazala
66824779d892ee6886af5da9
Wenyu Liu
66824779d892ee6886af5daa
Xinggang Wang
admin_assigned
2024-07-01T09:42:04.985Z
62600de6d47e3dbae32ce1ce
/avatars/a536417cfec6e10ac415091bd1829426.svg
Xinggang Wang
user
xinggangw
EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything
  Model
Segment Anything Model (SAM) has attracted widespread attention for its
superior interactive segmentation capabilities with visual prompts while
lacking further exploration of text prompts. In this paper, we empirically
investigate what text prompt encoders (e.g., CLIP or LLM) are good for adapting
SAM for referring expression segmentation and introduce the Early
Vision-language Fusion-based SAM (EVF-SAM). EVF-SAM is a simple yet effective
referring segmentation method which exploits multimodal prompts (i.e., image
and text) and comprises a pre-trained vision-language model to generate
referring prompts and a SAM model for segmentation. Surprisingly, we observe
that: (1) multimodal prompts and (2) vision-language models with early fusion
(e.g., BEIT-3) are beneficial for prompting SAM for accurate referring
segmentation. Our experiments show that the proposed EVF-SAM based on BEIT-3
can obtain state-of-the-art performance on RefCOCO/+/g for referring expression
segmentation and demonstrate the superiority of prompting SAM with early
vision-language fusion. In addition, the proposed EVF-SAM with 1.32B parameters
achieves remarkably higher performance while reducing nearly 82% of parameters
compared to previous SAM methods based on large multimodal models.
EVF-SAM combines image and text prompts with early vision-language fusion for improved referring expression segmentation, achieving state-of-the-art results on RefCOCO/+/g with fewer parameters.
referring expression segmentation text prompt encoders CLIP LLM SAM multimodal prompts pre-trained vision-language model BEIT-3 early vision-language fusion parameter-efficient fine-tuning