Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept
  Customization of Diffusion Models
Public large-scale text-to-image diffusion models, such as Stable Diffusion,
have gained significant attention from the community. These models can be
easily customized for new concepts using low-rank adaptations (LoRAs). However,
the utilization of multiple concept LoRAs to jointly support multiple
customized concepts presents a challenge. We refer to this scenario as
decentralized multi-concept customization, which involves single-client concept
tuning and center-node concept fusion. In this paper, we propose a new
framework called Mix-of-Show that addresses the challenges of decentralized
multi-concept customization, including concept conflicts resulting from
existing single-client LoRA tuning and identity loss during model fusion.
Mix-of-Show adopts an embedding-decomposed LoRA (ED-LoRA) for single-client
tuning and gradient fusion for the center node to preserve the in-domain
essence of single concepts and support theoretically limitless concept fusion.
Additionally, we introduce regionally controllable sampling, which extends
spatially controllable sampling (e.g., ControlNet and T2I-Adaptor) to address
attribute binding and missing object problems in multi-concept sampling.
Extensive experiments demonstrate that Mix-of-Show is capable of composing
multiple customized concepts with high fidelity, including characters, objects,
and scenes.
64756f8122b4dec4524c5614
Yuchao Gu
claimed_verified
2023-05-30T09:01:32.114Z
63021630a35b21bd8a53305a
/avatars/7a7e8b39749eda61e57d8a1908726558.svg
Gu Yuchao
user
guyuchao
64756f8122b4dec4524c5615
Xintao Wang
claimed_verified
2024-01-18T08:21:29.663Z
60e272ca6c78a8c122b12127
https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
Xintao Wang
user
Xintao
64756f8122b4dec4524c5616
Jay Zhangjie Wu
claimed_verified
2023-10-16T07:25:28.335Z
633aaf695df91da9cea92960
https://cdn-avatars.huggingface.co/v1/production/uploads/633aaf695df91da9cea92960/h1oOWUudbdlsd63Q5w0hM.png
Jay Wu
user
jayw
64756f8122b4dec4524c5617
Yujun Shi
claimed_verified
2023-07-10T09:25:23.067Z
645ce309a03f3ebb0bdf2b89
/avatars/652db4d2badd1191c6612f0950a88a88.svg
Yujun Shi
user
PeiyangShi
64756f8122b4dec4524c5618
Yunpeng Chen
64756f8122b4dec4524c5619
Zihan Fan
64756f8122b4dec4524c561a
Wuyou Xiao
64756f8122b4dec4524c561b
Rui Zhao
claimed_verified
2024-10-14T07:49:27.060Z
652b83b73b5997ed71a310f2
https://cdn-avatars.huggingface.co/v1/production/uploads/652b83b73b5997ed71a310f2/ipCpdeHUp4-0OmRz5z8IW.png
Rui Zhao
user
ruizhaocv
64756f8122b4dec4524c561c
Shuning Chang
64756f8122b4dec4524c561d
Weijia Wu
64756f8122b4dec4524c561e
Yixiao Ge
claimed_verified
2023-06-01T16:30:43.920Z
640e9762b03f4cd29f58d982
/avatars/81da37d628163fe3e094b247c7c3a3b5.svg
Yixiao Ge
user
yxgeee
64756f8122b4dec4524c561f
Ying Shan
64756f8122b4dec4524c5620
Mike Zheng Shou
Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept
  Customization of Diffusion Models
Public large-scale text-to-image diffusion models, such as Stable Diffusion,
have gained significant attention from the community. These models can be
easily customized for new concepts using low-rank adaptations (LoRAs). However,
the utilization of multiple concept LoRAs to jointly support multiple
customized concepts presents a challenge. We refer to this scenario as
decentralized multi-concept customization, which involves single-client concept
tuning and center-node concept fusion. In this paper, we propose a new
framework called Mix-of-Show that addresses the challenges of decentralized
multi-concept customization, including concept conflicts resulting from
existing single-client LoRA tuning and identity loss during model fusion.
Mix-of-Show adopts an embedding-decomposed LoRA (ED-LoRA) for single-client
tuning and gradient fusion for the center node to preserve the in-domain
essence of single concepts and support theoretically limitless concept fusion.
Additionally, we introduce regionally controllable sampling, which extends
spatially controllable sampling (e.g., ControlNet and T2I-Adaptor) to address
attribute binding and missing object problems in multi-concept sampling.
Extensive experiments demonstrate that Mix-of-Show is capable of composing
multiple customized concepts with high fidelity, including characters, objects,
and scenes.
Mix-of-Show addresses challenges in decentralized multi-concept customization for text-to-image diffusion models using embedding-decomposed LoRA, gradient fusion, and regionally controllable sampling.
text-to-image diffusion models Stable Diffusion low-rank adaptations LoRAs decentralized multi-concept customization single-client concept tuning center-node concept fusion concept conflicts identity loss embedding-decomposed LoRA gradient fusion regionally controllable sampling ControlNet T2I-Adaptor attribute binding missing object problems