The Hidden Language of Diffusion Models
Text-to-image diffusion models have demonstrated an unparalleled ability to
generate high-quality, diverse images from a textual concept (e.g., "a doctor",
"love"). However, the internal process of mapping text to a rich visual
representation remains an enigma. In this work, we tackle the challenge of
understanding concept representations in text-to-image models by decomposing an
input text prompt into a small set of interpretable elements. This is achieved
by learning a pseudo-token that is a sparse weighted combination of tokens from
the model's vocabulary, with the objective of reconstructing the images
generated for the given concept. Applied over the state-of-the-art Stable
Diffusion model, this decomposition reveals non-trivial and surprising
structures in the representations of concepts. For example, we find that some
concepts such as "a president" or "a composer" are dominated by specific
instances (e.g., "Obama", "Biden") and their interpolations. Other concepts,
such as "happiness" combine associated terms that can be concrete ("family",
"laughter") or abstract ("friendship", "emotion"). In addition to peering into
the inner workings of Stable Diffusion, our method also enables applications
such as single-image decomposition to tokens, bias detection and mitigation,
and semantic image manipulation. Our code will be available at:
https://hila-chefer.github.io/Conceptor/
64796c09e504380e5c896730
Hila Chefer
claimed_verified
2023-06-02T07:16:50.696Z
6181c72cdcc1df2c9de8a4d8
https://cdn-avatars.huggingface.co/v1/production/uploads/1655248010394-6181c72cdcc1df2c9de8a4d8.jpeg
Hila Chefer
user
Hila
64796c09e504380e5c896731
Oran Lang
64796c09e504380e5c896732
Mor Geva
admin_assigned
2023-06-05T12:03:58.348Z
610b729f9da682cd54ad9adf
https://cdn-avatars.huggingface.co/v1/production/uploads/1628140189042-noauth.jpeg
Mor Geva
user
mega
64796c09e504380e5c896733
Volodymyr Polosukhin
64796c09e504380e5c896734
Assaf Shocher
64796c09e504380e5c896735
Michal Irani
64796c09e504380e5c896736
Inbar Mosseri
64796c09e504380e5c896737
Lior Wolf
admin_assigned
2023-06-05T12:04:29.656Z
62f9f170363251ee40a08046
/avatars/67ad459db44b605be0c1f6bd33363d6e.svg
Lior Wolf
user
liorwolf
The Hidden Language of Diffusion Models
Text-to-image diffusion models have demonstrated an unparalleled ability to
generate high-quality, diverse images from a textual concept (e.g., "a doctor",
"love"). However, the internal process of mapping text to a rich visual
representation remains an enigma. In this work, we tackle the challenge of
understanding concept representations in text-to-image models by decomposing an
input text prompt into a small set of interpretable elements. This is achieved
by learning a pseudo-token that is a sparse weighted combination of tokens from
the model's vocabulary, with the objective of reconstructing the images
generated for the given concept. Applied over the state-of-the-art Stable
Diffusion model, this decomposition reveals non-trivial and surprising
structures in the representations of concepts. For example, we find that some
concepts such as "a president" or "a composer" are dominated by specific
instances (e.g., "Obama", "Biden") and their interpolations. Other concepts,
such as "happiness" combine associated terms that can be concrete ("family",
"laughter") or abstract ("friendship", "emotion"). In addition to peering into
the inner workings of Stable Diffusion, our method also enables applications
such as single-image decomposition to tokens, bias detection and mitigation,
and semantic image manipulation. Our code will be available at:
https://hila-chefer.github.io/Conceptor/
A method for decomposing text prompts into interpretable elements within text-to-image diffusion models reveals underlying structures and enables applications like bias detection and semantic manipulation.
text-to-image diffusion models Stable Diffusion pseudo-token sparse weighted combination concept representation concept decomposition single-image decomposition bias detection semantic image manipulation