SLiMe: Segment Like Me
Significant strides have been made using large vision-language models, like
Stable Diffusion (SD), for a variety of downstream tasks, including image
editing, image correspondence, and 3D shape generation. Inspired by these
advancements, we explore leveraging these extensive vision-language models for
segmenting images at any desired granularity using as few as one annotated
sample by proposing SLiMe. SLiMe frames this problem as an optimization task.
Specifically, given a single training image and its segmentation mask, we first
extract attention maps, including our novel "weighted accumulated
self-attention map" from the SD prior. Then, using the extracted attention
maps, the text embeddings of Stable Diffusion are optimized such that, each of
them, learn about a single segmented region from the training image. These
learned embeddings then highlight the segmented region in the attention maps,
which in turn can then be used to derive the segmentation map. This enables
SLiMe to segment any real-world image during inference with the granularity of
the segmented region in the training image, using just one example. Moreover,
leveraging additional training data when available, i.e. few-shot, improves the
performance of SLiMe. We carried out a knowledge-rich set of experiments
examining various design factors and showed that SLiMe outperforms other
existing one-shot and few-shot segmentation methods.
64f9321fb2c85cf9e521c01c
Aliasghar Khani
claimed_verified
2023-09-07T06:55:42.566Z
6372b048f7e99092f9066683
/avatars/e78eb302089da7e5073f9a8f4118e2d7.svg
aliasghar
user
aka225
64f9321fb2c85cf9e521c01d
Saeid Asgari Taghanaki
64f9321fb2c85cf9e521c01e
Aditya Sanghi
64f9321fb2c85cf9e521c01f
Ali Mahdavi Amiri
admin_assigned
2023-09-07T12:54:41.364Z
64233145d654afd693193a17
/avatars/9c668059e68773fab120f1a12c82f3ed.svg
Ali Mahdavi-Amiri
user
amahdavi
64f9321fb2c85cf9e521c020
Ghassan Hamarneh
claimed_verified
2023-09-07T13:23:37.831Z
64f9cc8ad56420ff571ab35c
/avatars/87085c7f4ca3a0c30b64c4486b89353c.svg
Ghassan Hamarneh
user
Hamarneh
SLiMe: Segment Like Me
Significant strides have been made using large vision-language models, like
Stable Diffusion (SD), for a variety of downstream tasks, including image
editing, image correspondence, and 3D shape generation. Inspired by these
advancements, we explore leveraging these extensive vision-language models for
segmenting images at any desired granularity using as few as one annotated
sample by proposing SLiMe. SLiMe frames this problem as an optimization task.
Specifically, given a single training image and its segmentation mask, we first
extract attention maps, including our novel "weighted accumulated
self-attention map" from the SD prior. Then, using the extracted attention
maps, the text embeddings of Stable Diffusion are optimized such that, each of
them, learn about a single segmented region from the training image. These
learned embeddings then highlight the segmented region in the attention maps,
which in turn can then be used to derive the segmentation map. This enables
SLiMe to segment any real-world image during inference with the granularity of
the segmented region in the training image, using just one example. Moreover,
leveraging additional training data when available, i.e. few-shot, improves the
performance of SLiMe. We carried out a knowledge-rich set of experiments
examining various design factors and showed that SLiMe outperforms other
existing one-shot and few-shot segmentation methods.
SLiMe segments images at desired granularity using Stable Diffusion with minimal annotations and outperforms existing one-shot and few-shot segmentation methods.
Stable Diffusion attention maps weighted accumulated self-attention map text embeddings optimization task one-shot segmentation few-shot segmentation