Customizing Text-to-Image Models with a Single Image Pair
Art reinterpretation is the practice of creating a variation of a reference
work, making a paired artwork that exhibits a distinct artistic style. We ask
if such an image pair can be used to customize a generative model to capture
the demonstrated stylistic difference. We propose Pair Customization, a new
customization method that learns stylistic difference from a single image pair
and then applies the acquired style to the generation process. Unlike existing
methods that learn to mimic a single concept from a collection of images, our
method captures the stylistic difference between paired images. This allows us
to apply a stylistic change without overfitting to the specific image content
in the examples. To address this new task, we employ a joint optimization
method that explicitly separates the style and content into distinct LoRA
weight spaces. We optimize these style and content weights to reproduce the
style and content images while encouraging their orthogonality. During
inference, we modify the diffusion process via a new style guidance based on
our learned weights. Both qualitative and quantitative experiments show that
our method can effectively learn style while avoiding overfitting to image
content, highlighting the potential of modeling such stylistic differences from
a single image pair.
66345f93488d269f8f82a068
Maxwell Jones
admin_assigned
2024-05-03T07:49:31.664Z
634e2e38be5a827d4875fd99
https://cdn-avatars.huggingface.co/v1/production/uploads/1666068012901-noauth.png
Maxwell Jones
user
maxwelljones14
66345f93488d269f8f82a069
Sheng-Yu Wang
admin_assigned
2024-05-03T07:49:38.385Z
62ecaeb1d0d93142d672a067
/avatars/46a572438c4ac0a0b0d24c7033b8cb53.svg
Sheng-Yu Wang
user
sywang
66345f93488d269f8f82a06a
Nupur Kumari
admin_assigned
2024-05-03T07:49:50.368Z
62f6a894c3372328414c7021
/avatars/e8b10912355712f38f10805c31bea962.svg
Nupur Kumari
user
nupurkmr9
66345f93488d269f8f82a06b
David Bau
admin_assigned
2024-05-03T07:50:18.179Z
6214d6c01e35c843d42d1f77
/avatars/ac208cd180b4f3ed1ec367e581facfcf.svg
David Bau
user
davidbau
66345f93488d269f8f82a06c
Jun-Yan Zhu
admin_assigned
2024-05-03T07:50:25.222Z
63a0acc32fabbbb899952a2b
https://cdn-avatars.huggingface.co/v1/production/uploads/1671474335794-noauth.jpeg
Jun-Yan Zhu
user
junyanz
Customizing Text-to-Image Models with a Single Image Pair
Art reinterpretation is the practice of creating a variation of a reference
work, making a paired artwork that exhibits a distinct artistic style. We ask
if such an image pair can be used to customize a generative model to capture
the demonstrated stylistic difference. We propose Pair Customization, a new
customization method that learns stylistic difference from a single image pair
and then applies the acquired style to the generation process. Unlike existing
methods that learn to mimic a single concept from a collection of images, our
method captures the stylistic difference between paired images. This allows us
to apply a stylistic change without overfitting to the specific image content
in the examples. To address this new task, we employ a joint optimization
method that explicitly separates the style and content into distinct LoRA
weight spaces. We optimize these style and content weights to reproduce the
style and content images while encouraging their orthogonality. During
inference, we modify the diffusion process via a new style guidance based on
our learned weights. Both qualitative and quantitative experiments show that
our method can effectively learn style while avoiding overfitting to image
content, highlighting the potential of modeling such stylistic differences from
a single image pair.
A new method called Pair Customization learns and applies stylistic differences from a single image pair to a generative model without overfitting to specific image content.
Pair Customization stylistic difference single image pair generative model joint optimization LoRA weight spaces orthogonality diffusion process style guidance