ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs
Methods for finetuning generative models for concept-driven personalization
generally achieve strong results for subject-driven or style-driven generation.
Recently, low-rank adaptations (LoRA) have been proposed as a
parameter-efficient way of achieving concept-driven personalization. While
recent work explores the combination of separate LoRAs to achieve joint
generation of learned styles and subjects, existing techniques do not reliably
address the problem; they often compromise either subject fidelity or style
fidelity. We propose ZipLoRA, a method to cheaply and effectively merge
independently trained style and subject LoRAs in order to achieve generation of
any user-provided subject in any user-provided style. Experiments on a wide
range of subject and style combinations show that ZipLoRA can generate
compelling results with meaningful improvements over baselines in subject and
style fidelity while preserving the ability to recontextualize. Project page:
https://ziplora.github.io
655ef0c7337a5d2e37b08c23
Viraj Shah
admin_assigned
2023-11-23T10:55:03.035Z
645c1b97081be4b320478149
/avatars/d2714416c008c51feb11ea2e67cdc98e.svg
Viraj Shah
user
shahviraj
655ef0c7337a5d2e37b08c24
Nataniel Ruiz
claimed_verified
2023-12-05T08:41:42.775Z
6318c6e6cb116eab31d70a16
https://cdn-avatars.huggingface.co/v1/production/uploads/6318c6e6cb116eab31d70a16/yAw5XtvqaPUkPj2B6JOzz.jpeg
Nataniel
user
flavoredquark
655ef0c7337a5d2e37b08c25
Forrester Cole
admin_assigned
2023-11-23T10:54:08.066Z
635003011d81beb8e244e68d
/avatars/32f46e6778d1c52ebdd6f2511ba48257.svg
Forrester Cole
user
fcole
655ef0c7337a5d2e37b08c26
Erika Lu
655ef0c7337a5d2e37b08c27
Svetlana Lazebnik
655ef0c7337a5d2e37b08c28
Yuanzhen Li
admin_assigned
2023-11-23T10:54:27.631Z
6334ee5831a2be3938c85639
/avatars/9821af26a585f54b595ec39f061f2250.svg
Yuanzhen Li
user
yzli
655ef0c7337a5d2e37b08c29
Varun Jampani
admin_assigned
2023-11-23T10:54:35.759Z
630e7cfe3fc17ffc50f6e835
/avatars/0b742ff094a09f9374fafcd97ab9e002.svg
Varun Jampani
user
varunjampani
ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs
Methods for finetuning generative models for concept-driven personalization
generally achieve strong results for subject-driven or style-driven generation.
Recently, low-rank adaptations (LoRA) have been proposed as a
parameter-efficient way of achieving concept-driven personalization. While
recent work explores the combination of separate LoRAs to achieve joint
generation of learned styles and subjects, existing techniques do not reliably
address the problem; they often compromise either subject fidelity or style
fidelity. We propose ZipLoRA, a method to cheaply and effectively merge
independently trained style and subject LoRAs in order to achieve generation of
any user-provided subject in any user-provided style. Experiments on a wide
range of subject and style combinations show that ZipLoRA can generate
compelling results with meaningful improvements over baselines in subject and
style fidelity while preserving the ability to recontextualize. Project page:
https://ziplora.github.io
ZipLoRA effectively combines independently trained style and subject LoRAs to enhance both subject and style fidelity in generative models.
low-rank adaptations LoRA parameter-efficient ZipLoRA generation subject fidelity style fidelity recontextualize