Subject-Diffusion:Open Domain Personalized Text-to-Image Generation
  without Test-time Fine-tuning
Recent progress in personalized image generation using diffusion models has
been significant. However, development in the area of open-domain and
non-fine-tuning personalized image generation is proceeding rather slowly. In
this paper, we propose Subject-Diffusion, a novel open-domain personalized
image generation model that, in addition to not requiring test-time
fine-tuning, also only requires a single reference image to support
personalized generation of single- or multi-subject in any domain. Firstly, we
construct an automatic data labeling tool and use the LAION-Aesthetics dataset
to construct a large-scale dataset consisting of 76M images and their
corresponding subject detection bounding boxes, segmentation masks and text
descriptions. Secondly, we design a new unified framework that combines text
and image semantics by incorporating coarse location and fine-grained reference
image control to maximize subject fidelity and generalization. Furthermore, we
also adopt an attention control mechanism to support multi-subject generation.
Extensive qualitative and quantitative results demonstrate that our method
outperforms other SOTA frameworks in single, multiple, and human customized
image generation. Please refer to our
https://oppo-mente-lab.github.io/subject_diffusion/{project page}
64bdd9674b4ff0d5095be4b5
Jian Ma
admin_assigned
2023-07-24T07:56:03.795Z
63043115a362e7e8b51f47fa
/avatars/d5c034262ddfcc3e7231b9a5abf461ad.svg
majian
user
majian0318
64bdd9674b4ff0d5095be4b6
Junhao Liang
admin_assigned
2023-07-24T07:55:16.747Z
63426b3aaa45ed8ecf8fa9a8
/avatars/529886aac934835b79e089580a1f0553.svg
Junhao Liang
user
Leeeung
64bdd9674b4ff0d5095be4b7
Chen Chen
claimed_verified
2023-07-24T14:15:21.318Z
649d47e06503630c5269d93a
/avatars/6812c9f79f2bd753ed68afb03baac202.svg
Chen Chen
user
beckhamchen
64bdd9674b4ff0d5095be4b8
Haonan Lu
Subject-Diffusion:Open Domain Personalized Text-to-Image Generation
  without Test-time Fine-tuning
Recent progress in personalized image generation using diffusion models has
been significant. However, development in the area of open-domain and
non-fine-tuning personalized image generation is proceeding rather slowly. In
this paper, we propose Subject-Diffusion, a novel open-domain personalized
image generation model that, in addition to not requiring test-time
fine-tuning, also only requires a single reference image to support
personalized generation of single- or multi-subject in any domain. Firstly, we
construct an automatic data labeling tool and use the LAION-Aesthetics dataset
to construct a large-scale dataset consisting of 76M images and their
corresponding subject detection bounding boxes, segmentation masks and text
descriptions. Secondly, we design a new unified framework that combines text
and image semantics by incorporating coarse location and fine-grained reference
image control to maximize subject fidelity and generalization. Furthermore, we
also adopt an attention control mechanism to support multi-subject generation.
Extensive qualitative and quantitative results demonstrate that our method
outperforms other SOTA frameworks in single, multiple, and human customized
image generation. Please refer to our
https://oppo-mente-lab.github.io/subject_diffusion/{project page}
Subject-Diffusion, a novel open-domain personalized image generation model, uses a unified framework combining text and image semantics, coarse location control, fine-grained reference image control, and attention mechanism to generate accurate and flexible images without the need for fine-tuning or multiple reference images.
diffusion models Subject-Diffusion LAION-Aesthetics dataset subject detection bounding boxes segmentation masks text descriptions unified framework automatic data labeling tool text and image semantics coarse location control fine-grained reference image control attention control mechanism multi-subject generation human customized image generation