DiffuseKronA: A Parameter Efficient Fine-tuning Method for Personalized
  Diffusion Model
In the realm of subject-driven text-to-image (T2I) generative models, recent
developments like DreamBooth and BLIP-Diffusion have led to impressive results
yet encounter limitations due to their intensive fine-tuning demands and
substantial parameter requirements. While the low-rank adaptation (LoRA) module
within DreamBooth offers a reduction in trainable parameters, it introduces a
pronounced sensitivity to hyperparameters, leading to a compromise between
parameter efficiency and the quality of T2I personalized image synthesis.
Addressing these constraints, we introduce \textit{DiffuseKronA}, a
novel Kronecker product-based adaptation module that not only significantly
reduces the parameter count by 35\% and 99.947\% compared to LoRA-DreamBooth
and the original DreamBooth, respectively, but also enhances the quality of
image synthesis. Crucially, DiffuseKronA mitigates the issue of
hyperparameter sensitivity, delivering consistent high-quality generations
across a wide range of hyperparameters, thereby diminishing the necessity for
extensive fine-tuning. Furthermore, a more controllable decomposition makes
DiffuseKronA more interpretable and even can achieve up to a 50\%
reduction with results comparable to LoRA-Dreambooth. Evaluated against diverse
and complex input images and text prompts, DiffuseKronA consistently
outperforms existing models, producing diverse images of higher quality with
improved fidelity and a more accurate color distribution of objects, all the
while upholding exceptional parameter efficiency, thus presenting a substantial
advancement in the field of T2I generative modeling. Our project page,
consisting of links to the code, and pre-trained checkpoints, is available at
https://diffusekrona.github.io/{https://diffusekrona.github.io/}.
65ded73907334f484a6d53ff
Shyam Marjit
admin_assigned
2024-02-28T12:14:40.002Z
6400fde28ba76abe4b6f8af6
https://cdn-avatars.huggingface.co/v1/production/uploads/6400fde28ba76abe4b6f8af6/UTIl4ygTLwpDBDloY3ldW.jpeg
Shyam Marjit
user
ShyamMarjit
65ded73907334f484a6d5400
Harshit Singh
claimed_verified
2024-03-15T08:05:56.586Z
646e3c668dc026869caf69f0
/avatars/f87a452e7e4b048148b10361e6cdbf3f.svg
Harshit Singh
user
Harshan14
65ded73907334f484a6d5401
Nityanand Mathur
claimed_verified
2024-02-28T12:57:52.398Z
635c0de2cb0f36a40bb96fce
https://cdn-avatars.huggingface.co/v1/production/uploads/635c0de2cb0f36a40bb96fce/pqURAHT6eWbaHDWM-YJbM.jpeg
Nityanand Mathur
user
nityanandmathur
65ded73907334f484a6d5402
Sayak Paul
admin_assigned
2024-02-28T12:15:39.061Z
5f7fbd813e94f16a85448745
https://cdn-avatars.huggingface.co/v1/production/uploads/1649681653581-5f7fbd813e94f16a85448745.jpeg
Sayak Paul
user
sayakpaul
65ded73907334f484a6d5403
Chia-Mu Yu
65ded73907334f484a6d5404
Pin-Yu Chen
admin_assigned
2024-02-28T12:15:54.986Z
6495dd0b71f6708e0f990032
https://cdn-avatars.huggingface.co/v1/production/uploads/6495dd0b71f6708e0f990032/PBIjdKNnpkxvR_3djCGVm.png
Pin-Yu Chen
user
pinyuchen
DiffuseKronA: A Parameter Efficient Fine-tuning Method for Personalized
  Diffusion Model
In the realm of subject-driven text-to-image (T2I) generative models, recent
developments like DreamBooth and BLIP-Diffusion have led to impressive results
yet encounter limitations due to their intensive fine-tuning demands and
substantial parameter requirements. While the low-rank adaptation (LoRA) module
within DreamBooth offers a reduction in trainable parameters, it introduces a
pronounced sensitivity to hyperparameters, leading to a compromise between
parameter efficiency and the quality of T2I personalized image synthesis.
Addressing these constraints, we introduce \textit{DiffuseKronA}, a
novel Kronecker product-based adaptation module that not only significantly
reduces the parameter count by 35\% and 99.947\% compared to LoRA-DreamBooth
and the original DreamBooth, respectively, but also enhances the quality of
image synthesis. Crucially, DiffuseKronA mitigates the issue of
hyperparameter sensitivity, delivering consistent high-quality generations
across a wide range of hyperparameters, thereby diminishing the necessity for
extensive fine-tuning. Furthermore, a more controllable decomposition makes
DiffuseKronA more interpretable and even can achieve up to a 50\%
reduction with results comparable to LoRA-Dreambooth. Evaluated against diverse
and complex input images and text prompts, DiffuseKronA consistently
outperforms existing models, producing diverse images of higher quality with
improved fidelity and a more accurate color distribution of objects, all the
while upholding exceptional parameter efficiency, thus presenting a substantial
advancement in the field of T2I generative modeling. Our project page,
consisting of links to the code, and pre-trained checkpoints, is available at
https://diffusekrona.github.io/{https://diffusekrona.github.io/}.
DiffuseKronA, a Kronecker product-based adaptation module, reduces parameter counts and enhances quality in text-to-image generative models compared to LoRA and DreamBooth.
subject-driven text-to-image DreamBooth BLIP-Diffusion low-rank adaptation Kronecker product DiffuseKronA hyperparameter sensitivity parameter efficiency image synthesis interpretability