TweedieMix: Improving Multi-Concept Fusion for Diffusion-based
  Image/Video Generation
Despite significant advancements in customizing text-to-image and video
generation models, generating images and videos that effectively integrate
multiple personalized concepts remains a challenging task. To address this, we
present TweedieMix, a novel method for composing customized diffusion models
during the inference phase. By analyzing the properties of reverse diffusion
sampling, our approach divides the sampling process into two stages. During the
initial steps, we apply a multiple object-aware sampling technique to ensure
the inclusion of the desired target objects. In the later steps, we blend the
appearances of the custom concepts in the de-noised image space using Tweedie's
formula. Our results demonstrate that TweedieMix can generate multiple
personalized concepts with higher fidelity than existing methods. Moreover, our
framework can be effortlessly extended to image-to-video diffusion models,
enabling the generation of videos that feature multiple personalized concepts.
Results and source code are in our anonymous project page.
670759004341dcee45a674a6
Gihyun Kwon
extracted_confirmed
2024-10-10T04:36:37.757Z
66c68b917f4accbaaaaff6cc
/avatars/40f1a25c393f3cb2e7ae906bb954edac.svg
Gihyun Kwon
user
gkwon
670759004341dcee45a674a7
Jong Chul Ye
TweedieMix: Improving Multi-Concept Fusion for Diffusion-based
  Image/Video Generation
Despite significant advancements in customizing text-to-image and video
generation models, generating images and videos that effectively integrate
multiple personalized concepts remains a challenging task. To address this, we
present TweedieMix, a novel method for composing customized diffusion models
during the inference phase. By analyzing the properties of reverse diffusion
sampling, our approach divides the sampling process into two stages. During the
initial steps, we apply a multiple object-aware sampling technique to ensure
the inclusion of the desired target objects. In the later steps, we blend the
appearances of the custom concepts in the de-noised image space using Tweedie's
formula. Our results demonstrate that TweedieMix can generate multiple
personalized concepts with higher fidelity than existing methods. Moreover, our
framework can be effortlessly extended to image-to-video diffusion models,
enabling the generation of videos that feature multiple personalized concepts.
Results and source code are in our anonymous project page.
TweedieMix uses a two-stage diffusion sampling approach to effectively integrate multiple personalized concepts in image and video generation.
custom diffusion models reverse diffusion sampling multiple object-aware sampling Tweedie's formula image-to-video diffusion models