Text-to-Sticker: Style Tailoring Latent Diffusion Models for Human
  Expression
We introduce Style Tailoring, a recipe to finetune Latent Diffusion Models
(LDMs) in a distinct domain with high visual quality, prompt alignment and
scene diversity. We choose sticker image generation as the target domain, as
the images significantly differ from photorealistic samples typically generated
by large-scale LDMs. We start with a competent text-to-image model, like Emu,
and show that relying on prompt engineering with a photorealistic model to
generate stickers leads to poor prompt alignment and scene diversity. To
overcome these drawbacks, we first finetune Emu on millions of sticker-like
images collected using weak supervision to elicit diversity. Next, we curate
human-in-the-loop (HITL) Alignment and Style datasets from model generations,
and finetune to improve prompt alignment and style alignment respectively.
Sequential finetuning on these datasets poses a tradeoff between better style
alignment and prompt alignment gains. To address this tradeoff, we propose a
novel fine-tuning method called Style Tailoring, which jointly fits the content
and style distribution and achieves best tradeoff. Evaluation results show our
method improves visual quality by 14%, prompt alignment by 16.2% and scene
diversity by 15.3%, compared to prompt engineering the base Emu model for
stickers generation.
655c2fecc4bb0926a684c1f5
Animesh Sinha
claimed_verified
2023-11-21T10:24:22.674Z
63772d28e069bdcdbb8e4a29
/avatars/be71b00e009f2093a3ff79f07815e68e.svg
Animesh Sinha
user
animeshsinha
655c2fecc4bb0926a684c1f6
Bo Sun
claimed_verified
2023-11-22T09:48:36.015Z
6552a36bd04b38b354592bd5
https://cdn-avatars.huggingface.co/v1/production/uploads/6552a36bd04b38b354592bd5/KI8qpshurUOAkri-MKsm_.jpeg
Bo Sun
user
bsun0802
655c2fecc4bb0926a684c1f7
Anmol Kalia
claimed_verified
2023-12-11T08:16:48.968Z
6573c2a8769f3ee9bdf98a74
/avatars/df40b6153ec7368561049746a9c8adc2.svg
Anmol Kalia
user
anmolkalia
655c2fecc4bb0926a684c1f8
Arantxa Casanova
admin_assigned
2023-11-21T13:21:43.154Z
61798d543490eedabaa53911
/avatars/8cd5eef76ce7ed0f1efd3debad8a3d08.svg
Arantxa Casanova
user
acasanova
655c2fecc4bb0926a684c1f9
Elliot Blanchard
claimed_verified
2023-11-27T08:43:18.019Z
6563cf4a718b9db5215d07b4
/avatars/20e5be0bd3fcd888959a0af0f0b462e4.svg
Elliot Blanchard
user
elliotblanchard
655c2fecc4bb0926a684c1fa
David Yan
655c2fecc4bb0926a684c1fb
Winnie Zhang
admin_assigned
2023-11-21T13:22:07.935Z
64564b304a7ffb7d5a47f515
/avatars/b848f02609d9ee8ab7df350ccb274c23.svg
Winnie Zhang
user
WinnieJackJack
655c2fecc4bb0926a684c1fc
Tony Nelli
655c2fecc4bb0926a684c1fd
Jiahui Chen
655c2fecc4bb0926a684c1fe
Hardik Shah
655c2fecc4bb0926a684c1ff
Licheng Yu
admin_assigned
2023-11-21T13:23:49.720Z
6372b1f9bd81fae2b3a712e8
/avatars/bcdf7b540cc94882e0ae6035bb07a8f2.svg
Licheng Yu
user
lichengyu
655c2fecc4bb0926a684c200
Mitesh Kumar Singh
655c2fecc4bb0926a684c201
Ankit Ramchandani
655c2fecc4bb0926a684c202
Maziar Sanjabi
655c2fecc4bb0926a684c203
Sonal Gupta
admin_assigned
2023-11-21T13:24:40.879Z
634ecb814677a5891c0c7026
/avatars/a20cf325e4fe0a227ff80694ffea4a3d.svg
Sonal Gupta
user
thenextgenai
655c2fecc4bb0926a684c204
Amy Bearman
655c2fecc4bb0926a684c205
Dhruv Mahajan
admin_assigned
2023-11-21T13:24:50.780Z
64358a51f81a16e7436330f7
/avatars/71c350836e5d025b4c7b9f99f72c2e21.svg
Dhruv Mahajan
user
Dhruv2009
Text-to-Sticker: Style Tailoring Latent Diffusion Models for Human
  Expression
We introduce Style Tailoring, a recipe to finetune Latent Diffusion Models
(LDMs) in a distinct domain with high visual quality, prompt alignment and
scene diversity. We choose sticker image generation as the target domain, as
the images significantly differ from photorealistic samples typically generated
by large-scale LDMs. We start with a competent text-to-image model, like Emu,
and show that relying on prompt engineering with a photorealistic model to
generate stickers leads to poor prompt alignment and scene diversity. To
overcome these drawbacks, we first finetune Emu on millions of sticker-like
images collected using weak supervision to elicit diversity. Next, we curate
human-in-the-loop (HITL) Alignment and Style datasets from model generations,
and finetune to improve prompt alignment and style alignment respectively.
Sequential finetuning on these datasets poses a tradeoff between better style
alignment and prompt alignment gains. To address this tradeoff, we propose a
novel fine-tuning method called Style Tailoring, which jointly fits the content
and style distribution and achieves best tradeoff. Evaluation results show our
method improves visual quality by 14%, prompt alignment by 16.2% and scene
diversity by 15.3%, compared to prompt engineering the base Emu model for
stickers generation.
Style Tailoring is a finetuning method for Latent Diffusion Models that enhances the generation of sticker images, improving visual quality, prompt alignment, and scene diversity.
Latent Diffusion Models Style Tailoring sticker image generation text-to-image model Emu weak supervision human-in-the-loop prompt alignment style alignment