Gen4Gen: Generative Data Pipeline for Generative Multi-Concept
  Composition
Recent text-to-image diffusion models are able to learn and synthesize images
containing novel, personalized concepts (e.g., their own pets or specific
items) with just a few examples for training. This paper tackles two
interconnected issues within this realm of personalizing text-to-image
diffusion models. First, current personalization techniques fail to reliably
extend to multiple concepts -- we hypothesize this to be due to the mismatch
between complex scenes and simple text descriptions in the pre-training dataset
(e.g., LAION). Second, given an image containing multiple personalized
concepts, there lacks a holistic metric that evaluates performance on not just
the degree of resemblance of personalized concepts, but also whether all
concepts are present in the image and whether the image accurately reflects the
overall text description. To address these issues, we introduce Gen4Gen, a
semi-automated dataset creation pipeline utilizing generative models to combine
personalized concepts into complex compositions along with text-descriptions.
Using this, we create a dataset called MyCanvas, that can be used to benchmark
the task of multi-concept personalization. In addition, we design a
comprehensive metric comprising two scores (CP-CLIP and TI-CLIP) for better
quantifying the performance of multi-concept, personalized text-to-image
diffusion methods. We provide a simple baseline built on top of Custom
Diffusion with empirical prompting strategies for future researchers to
evaluate on MyCanvas. We show that by improving data quality and prompting
strategies, we can significantly increase multi-concept personalized image
generation quality, without requiring any modifications to model architecture
or training algorithms.
65dbf547f386d08eb0298b35
Chun-Hsiao Yeh
claimed_verified
2024-02-26T09:24:24.689Z
6499eca0685215f7247bd5ce
/avatars/b6fea0c33c3c930c7314b99b414072a9.svg
Chun-Hsiao Yeh
user
danielchyeh
65dbf547f386d08eb0298b36
Ta-Ying Cheng
admin_assigned
2024-02-26T10:20:41.984Z
6442b024e255a338677d78fa
/avatars/516769c6c3bc7a0d416924181c6866dc.svg
Ta-Ying Cheng
user
chengtim
65dbf547f386d08eb0298b37
He-Yen Hsieh
claimed_verified
2024-02-28T08:28:27.412Z
65de5db9f546556af598426a
/avatars/6f179ea76af9edc995d494517c3086ac.svg
Hsieh
user
He-Yen
65dbf547f386d08eb0298b38
Chuan-En Lin
admin_assigned
2024-02-26T10:20:54.997Z
617e396ba7481f8efe4c82fd
https://cdn-avatars.huggingface.co/v1/production/uploads/617e396ba7481f8efe4c82fd/SaxzURusTKUubf0sFazBd.png
David Chuan-En Lin
user
chuanenlin
65dbf547f386d08eb0298b39
Yi Ma
65dbf547f386d08eb0298b3a
Andrew Markham
admin_assigned
2024-02-26T10:21:53.883Z
63fca1aa9df782461c7c6367
/avatars/d61de110a4e87da80a5ca5820921773a.svg
Andrew Markham
user
markham
65dbf547f386d08eb0298b3b
Niki Trigoni
65dbf547f386d08eb0298b3c
H. T. Kung
65dbf547f386d08eb0298b3d
Yubei Chen
Gen4Gen: Generative Data Pipeline for Generative Multi-Concept
  Composition
Recent text-to-image diffusion models are able to learn and synthesize images
containing novel, personalized concepts (e.g., their own pets or specific
items) with just a few examples for training. This paper tackles two
interconnected issues within this realm of personalizing text-to-image
diffusion models. First, current personalization techniques fail to reliably
extend to multiple concepts -- we hypothesize this to be due to the mismatch
between complex scenes and simple text descriptions in the pre-training dataset
(e.g., LAION). Second, given an image containing multiple personalized
concepts, there lacks a holistic metric that evaluates performance on not just
the degree of resemblance of personalized concepts, but also whether all
concepts are present in the image and whether the image accurately reflects the
overall text description. To address these issues, we introduce Gen4Gen, a
semi-automated dataset creation pipeline utilizing generative models to combine
personalized concepts into complex compositions along with text-descriptions.
Using this, we create a dataset called MyCanvas, that can be used to benchmark
the task of multi-concept personalization. In addition, we design a
comprehensive metric comprising two scores (CP-CLIP and TI-CLIP) for better
quantifying the performance of multi-concept, personalized text-to-image
diffusion methods. We provide a simple baseline built on top of Custom
Diffusion with empirical prompting strategies for future researchers to
evaluate on MyCanvas. We show that by improving data quality and prompting
strategies, we can significantly increase multi-concept personalized image
generation quality, without requiring any modifications to model architecture
or training algorithms.
Gen4Gen addresses data and metric challenges in multi-concept personalization for text-to-image diffusion models, enhancing image generation quality through improved data creation and empirical prompting strategies.
text-to-image diffusion models personalized concepts MyCanvas CP-CLIP TI-CLIP Custom Diffusion empirical prompting strategies multi-concept personalization