Personalize Segment Anything Model with One Shot
Driven by large-data pre-training, Segment Anything Model (SAM) has been
demonstrated as a powerful and promptable framework, revolutionizing the
segmentation models. Despite the generality, customizing SAM for specific
visual concepts without man-powered prompting is under explored, e.g.,
automatically segmenting your pet dog in different images. In this paper, we
propose a training-free Personalization approach for SAM, termed as PerSAM.
Given only a single image with a reference mask, PerSAM first localizes the
target concept by a location prior, and segments it within other images or
videos via three techniques: target-guided attention, target-semantic
prompting, and cascaded post-refinement. In this way, we effectively adapt SAM
for private use without any training. To further alleviate the mask ambiguity,
we present an efficient one-shot fine-tuning variant, PerSAM-F. Freezing the
entire SAM, we introduce two learnable weights for multi-scale masks, only
training 2 parameters within 10 seconds for improved performance. To
demonstrate our efficacy, we construct a new segmentation dataset, PerSeg, for
personalized evaluation, and test our methods on video object segmentation with
competitive performance. Besides, our approach can also enhance DreamBooth to
personalize Stable Diffusion for text-to-image generation, which discards the
background disturbance for better target appearance learning. Code is released
at https://github.com/ZrrSkywalker/Personalize-SAM
645453d1bb73c0a97e992f60
Renrui Zhang
claimed_verified
2024-03-22T08:41:31.306Z
645b8b2687c79b6ec0bb3b7a
/avatars/00a9db32a42dc950112bf2593bb109cb.svg
Renrui
user
ZrrSkywalker
645453d1bb73c0a97e992f61
Zhengkai Jiang
claimed_verified
2023-05-08T09:27:08.683Z
6371dda2ffc0489ed7d712c2
/avatars/dcb80cae00535c22af99baf86cb70d93.svg
zkjiang
user
justin-zk
645453d1bb73c0a97e992f62
Ziyu Guo
645453d1bb73c0a97e992f63
Shilin Yan
645453d1bb73c0a97e992f64
Junting Pan
645453d1bb73c0a97e992f65
Hao Dong
645453d1bb73c0a97e992f66
Peng Gao
645453d1bb73c0a97e992f67
Hongsheng Li
claimed_verified
2024-02-05T10:51:15.583Z
65c04e9c27a5fdca81abcbd9
/avatars/12a155683c824fa23da4a9e2bed4f64e.svg
Hongsheng LI
user
hsli-cuhk
Personalize Segment Anything Model with One Shot
Driven by large-data pre-training, Segment Anything Model (SAM) has been
demonstrated as a powerful and promptable framework, revolutionizing the
segmentation models. Despite the generality, customizing SAM for specific
visual concepts without man-powered prompting is under explored, e.g.,
automatically segmenting your pet dog in different images. In this paper, we
propose a training-free Personalization approach for SAM, termed as PerSAM.
Given only a single image with a reference mask, PerSAM first localizes the
target concept by a location prior, and segments it within other images or
videos via three techniques: target-guided attention, target-semantic
prompting, and cascaded post-refinement. In this way, we effectively adapt SAM
for private use without any training. To further alleviate the mask ambiguity,
we present an efficient one-shot fine-tuning variant, PerSAM-F. Freezing the
entire SAM, we introduce two learnable weights for multi-scale masks, only
training 2 parameters within 10 seconds for improved performance. To
demonstrate our efficacy, we construct a new segmentation dataset, PerSeg, for
personalized evaluation, and test our methods on video object segmentation with
competitive performance. Besides, our approach can also enhance DreamBooth to
personalize Stable Diffusion for text-to-image generation, which discards the
background disturbance for better target appearance learning. Code is released
at https://github.com/ZrrSkywalker/Personalize-SAM
A training-free and fine-tuning variant of the Segment Anything Model (SAM), PerSAM and PerSAM-F, achieves personalized image and video segmentation using a single reference image and minimal fine-tuning, improving performance on personalized and dreambooth applications.
Segment Anything Model SAM PerSAM PerSAM-F target-guided attention target-semantic prompting cascaded post-refinement one-shot fine-tuning location prior PerSeg video object segmentation DreamBooth Stable Diffusion