Interpreting the Weight Space of Customized Diffusion Models
We investigate the space of weights spanned by a large collection of
customized diffusion models. We populate this space by creating a dataset of
over 60,000 models, each of which is a base model fine-tuned to insert a
different person's visual identity. We model the underlying manifold of these
weights as a subspace, which we term weights2weights. We demonstrate three
immediate applications of this space -- sampling, editing, and inversion.
First, as each point in the space corresponds to an identity, sampling a set of
weights from it results in a model encoding a novel identity. Next, we find
linear directions in this space corresponding to semantic edits of the identity
(e.g., adding a beard). These edits persist in appearance across generated
samples. Finally, we show that inverting a single image into this space
reconstructs a realistic identity, even if the input image is out of
distribution (e.g., a painting). Our results indicate that the weight space of
fine-tuned diffusion models behaves as an interpretable latent space of
identities.
666bae6c3214e94388ea7606
Amil Dravid
claimed_verified
2024-06-14T07:22:40.931Z
655e490d3beaa281e50aff5f
/avatars/7e6d9a096a71d601f66279e317d11e1f.svg
Amil Dravid
user
amildravid4292
666bae6c3214e94388ea7607
Yossi Gandelsman
admin_assigned
2024-06-14T09:57:15.404Z
62f6942205ca68c0e0ff7b97
/avatars/b0600de531b4c88f4aa7c30c5ceb06e1.svg
Yossi Gandelsman
user
yossig
666bae6c3214e94388ea7608
Kuan-Chieh Wang
admin_assigned
2024-06-14T09:57:20.946Z
648ca58a39d2584ee47efef6
https://cdn-avatars.huggingface.co/v1/production/uploads/648ca58a39d2584ee47efef6/R7B72bnwc59mdK45rmzYS.png
Kuan-Chieh Wang
user
wangkua1
666bae6c3214e94388ea7609
Rameen Abdal
admin_assigned
2024-06-14T09:57:26.935Z
630428fa7b50dd9d0a38cde0
/avatars/d1baf7fd17daf4be16ba5bd6cd4f2277.svg
Rameen Abdal
user
RameenAbdal
666bae6c3214e94388ea760a
Gordon Wetzstein
666bae6c3214e94388ea760b
Alexei A. Efros
666bae6c3214e94388ea760c
Kfir Aberman
Interpreting the Weight Space of Customized Diffusion Models
We investigate the space of weights spanned by a large collection of
customized diffusion models. We populate this space by creating a dataset of
over 60,000 models, each of which is a base model fine-tuned to insert a
different person's visual identity. We model the underlying manifold of these
weights as a subspace, which we term weights2weights. We demonstrate three
immediate applications of this space -- sampling, editing, and inversion.
First, as each point in the space corresponds to an identity, sampling a set of
weights from it results in a model encoding a novel identity. Next, we find
linear directions in this space corresponding to semantic edits of the identity
(e.g., adding a beard). These edits persist in appearance across generated
samples. Finally, we show that inverting a single image into this space
reconstructs a realistic identity, even if the input image is out of
distribution (e.g., a painting). Our results indicate that the weight space of
fine-tuned diffusion models behaves as an interpretable latent space of
identities.
A large dataset of fine-tuned diffusion models for different visual identities reveals a manifold representing identity in the weight space, enabling sampling, semantic editing, and inversion tasks.
diffusion models weights2weights sampling editing inversion