BootPIG: Bootstrapping Zero-shot Personalized Image Generation
  Capabilities in Pretrained Diffusion Models
Recent text-to-image generation models have demonstrated incredible success
in generating images that faithfully follow input prompts. However, the
requirement of using words to describe a desired concept provides limited
control over the appearance of the generated concepts. In this work, we address
this shortcoming by proposing an approach to enable personalization
capabilities in existing text-to-image diffusion models. We propose a novel
architecture (BootPIG) that allows a user to provide reference images of an
object in order to guide the appearance of a concept in the generated images.
  The proposed BootPIG architecture makes minimal modifications to a pretrained
text-to-image diffusion model and utilizes a separate UNet model to steer the
generations toward the desired appearance. We introduce a training procedure
that allows us to bootstrap personalization capabilities in the BootPIG
architecture using data generated from pretrained text-to-image models, LLM
chat agents, and image segmentation models. In contrast to existing methods
that require several days of pretraining, the BootPIG architecture can be
trained in approximately 1 hour. Experiments on the DreamBooth dataset
demonstrate that BootPIG outperforms existing zero-shot methods while being
comparable with test-time finetuning approaches. Through a user study, we
validate the preference for BootPIG generations over existing methods both in
maintaining fidelity to the reference object's appearance and aligning with
textual prompts.
65b3149af32c79fea798e6ec
Senthil Purushwalkam
admin_assigned
2024-01-26T10:17:43.719Z
637e633366a51ca5d960a9b1
/avatars/e53242e4e36875105be087c46a552e29.svg
Senthil Purushwalkam
user
senthilps8
65b3149af32c79fea798e6ed
Akash Gokul
admin_assigned
2024-01-26T10:17:49.780Z
63093856b105f8675bd1c783
/avatars/505a7b99893330a36a136d13fa99e05e.svg
Akash Gokul
user
akashgokul
65b3149af32c79fea798e6ee
Shafiq Joty
65b3149af32c79fea798e6ef
Nikhil Naik
admin_assigned
2024-01-26T10:18:04.969Z
627c2c751b02d7b5bf663fb0
/avatars/f33ac735e2cc910d2da1286d476de5db.svg
Nikhil Naik
user
nikdnaik
BootPIG: Bootstrapping Zero-shot Personalized Image Generation
  Capabilities in Pretrained Diffusion Models
Recent text-to-image generation models have demonstrated incredible success
in generating images that faithfully follow input prompts. However, the
requirement of using words to describe a desired concept provides limited
control over the appearance of the generated concepts. In this work, we address
this shortcoming by proposing an approach to enable personalization
capabilities in existing text-to-image diffusion models. We propose a novel
architecture (BootPIG) that allows a user to provide reference images of an
object in order to guide the appearance of a concept in the generated images.
  The proposed BootPIG architecture makes minimal modifications to a pretrained
text-to-image diffusion model and utilizes a separate UNet model to steer the
generations toward the desired appearance. We introduce a training procedure
that allows us to bootstrap personalization capabilities in the BootPIG
architecture using data generated from pretrained text-to-image models, LLM
chat agents, and image segmentation models. In contrast to existing methods
that require several days of pretraining, the BootPIG architecture can be
trained in approximately 1 hour. Experiments on the DreamBooth dataset
demonstrate that BootPIG outperforms existing zero-shot methods while being
comparable with test-time finetuning approaches. Through a user study, we
validate the preference for BootPIG generations over existing methods both in
maintaining fidelity to the reference object's appearance and aligning with
textual prompts.
BootPIG architecture enhances text-to-image generation models with personalization capabilities, using reference images, to improve the fidelity to the desired appearance while maintaining alignment with textual prompts.
text-to-image diffusion models BootPIG UNet DreamBooth zero-shot methods test-time finetuning