EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Models
Diffusion models have recently received increasing research attention for
their remarkable transfer abilities in semantic segmentation tasks. However,
generating fine-grained segmentation masks with diffusion models often requires
additional training on annotated datasets, leaving it unclear to what extent
pre-trained diffusion models alone understand the semantic relations of their
generated images. To address this question, we leverage the semantic knowledge
extracted from Stable Diffusion (SD) and aim to develop an image segmentor
capable of generating fine-grained segmentation maps without any additional
training. The primary difficulty stems from the fact that semantically
meaningful feature maps typically exist only in the spatially lower-dimensional
layers, which poses a challenge in directly extracting pixel-level semantic
relations from these feature maps. To overcome this issue, our framework
identifies semantic correspondences between image pixels and spatial locations
of low-dimensional feature maps by exploiting SD's generation process and
utilizes them for constructing image-resolution segmentation maps. In extensive
experiments, the produced segmentation maps are demonstrated to be well
delineated and capture detailed parts of the images, indicating the existence
of highly accurate pixel-level semantic knowledge in diffusion models.
65af54a6c92e09d5cdd088a2
Koichi Namekata
claimed_verified
2024-01-23T08:08:34.475Z
65aec2b6101482afcc3126f2
https://cdn-avatars.huggingface.co/v1/production/uploads/65aec2b6101482afcc3126f2/IEKT6u1Upx2clt7qhhoUB.jpeg
Koichi Namekata
user
kmcode
65af54a6c92e09d5cdd088a3
Amirmojtaba Sabour
claimed_verified
2024-01-23T17:09:20.093Z
656015f28138827138c70858
/avatars/b8471ae4d80f078c7c928fc3d8f49126.svg
Amirmojtaba Sabour
user
amsabour
65af54a6c92e09d5cdd088a4
Sanja Fidler
65af54a6c92e09d5cdd088a5
Seung Wook Kim
admin_assigned
2024-01-23T10:40:57.253Z
64fb5827d82fc6977d70582d
/avatars/54180874d2f162572c414970b8750759.svg
Seung Wook Kim
user
swkimbd
EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Models
Diffusion models have recently received increasing research attention for
their remarkable transfer abilities in semantic segmentation tasks. However,
generating fine-grained segmentation masks with diffusion models often requires
additional training on annotated datasets, leaving it unclear to what extent
pre-trained diffusion models alone understand the semantic relations of their
generated images. To address this question, we leverage the semantic knowledge
extracted from Stable Diffusion (SD) and aim to develop an image segmentor
capable of generating fine-grained segmentation maps without any additional
training. The primary difficulty stems from the fact that semantically
meaningful feature maps typically exist only in the spatially lower-dimensional
layers, which poses a challenge in directly extracting pixel-level semantic
relations from these feature maps. To overcome this issue, our framework
identifies semantic correspondences between image pixels and spatial locations
of low-dimensional feature maps by exploiting SD's generation process and
utilizes them for constructing image-resolution segmentation maps. In extensive
experiments, the produced segmentation maps are demonstrated to be well
delineated and capture detailed parts of the images, indicating the existence
of highly accurate pixel-level semantic knowledge in diffusion models.
Researchers develop a fine-grained image segmentor using Stable Diffusion without additional training, extracting semantic correspondences from low-dimensional feature maps for pixel-level segmentation.
diffusion models semantic segmentation tasks Stable Diffusion (SD) image segmentor fine-grained segmentation maps semantic knowledge low-dimensional feature maps semantic correspondences pixel-level semantic relations