ECLIPSE: A Resource-Efficient Text-to-Image Prior for Image Generations
Text-to-image (T2I) diffusion models, notably the unCLIP models (e.g.,
DALL-E-2), achieve state-of-the-art (SOTA) performance on various compositional
T2I benchmarks, at the cost of significant computational resources. The unCLIP
stack comprises T2I prior and diffusion image decoder. The T2I prior model
alone adds a billion parameters compared to the Latent Diffusion Models, which
increases the computational and high-quality data requirements. We introduce
ECLIPSE, a novel contrastive learning method that is both parameter and
data-efficient. ECLIPSE leverages pre-trained vision-language models (e.g.,
CLIP) to distill the knowledge into the prior model. We demonstrate that the
ECLIPSE trained prior, with only 3.3% of the parameters and trained on a mere
2.8% of the data, surpasses the baseline T2I priors with an average of 71.6%
preference score under resource-limited setting. It also attains performance on
par with SOTA big models, achieving an average of 63.36% preference score in
terms of the ability to follow the text compositions. Extensive experiments on
two unCLIP diffusion image decoders, Karlo and Kandinsky, affirm that ECLIPSE
priors consistently deliver high performance while significantly reducing
resource dependency.
6576712c1345577b70071a54
Maitreya Patel
claimed_verified
2023-12-11T08:15:44.751Z
622d2ff38d04fd29a9ccf1a7
https://cdn-avatars.huggingface.co/v1/production/uploads/622d2ff38d04fd29a9ccf1a7/ORpGrlU8Lm_oSEVftcZEK.jpeg
Maitreya Patel
user
mpatel57
6576712c1345577b70071a55
Changhoon Kim
admin_assigned
2023-12-11T08:19:42.622Z
634ad9bfd440b86afd4b045c
/avatars/3cce546fc4ad6a9c0bcf891b7ab6530b.svg
Changhoon Kim
user
fltr
6576712c1345577b70071a56
Sheng Cheng
claimed_verified
2023-12-11T08:15:41.752Z
63cca807336d9d56171486cf
/avatars/6d8b8c26f7b04e7b4ac313d095ab9e3e.svg
Sheng Cheng
user
duoduo1133
6576712c1345577b70071a57
Chitta Baral
admin_assigned
2023-12-11T08:19:49.555Z
62687430dc80d589a9f6cd45
/avatars/fe56b8c92657e4f210c86cc097d42813.svg
Chitta Baral
user
cogint-asu
6576712c1345577b70071a58
Yezhou Yang
claimed_verified
2023-12-12T08:09:17.694Z
6486a850903d5878eb98110f
https://cdn-avatars.huggingface.co/v1/production/uploads/6486a850903d5878eb98110f/ETSE52Fdq5kVPzy4R5tGt.jpeg
'YZ' Yezhou Yang
user
AiWithoutBorders
ECLIPSE: A Resource-Efficient Text-to-Image Prior for Image Generations
Text-to-image (T2I) diffusion models, notably the unCLIP models (e.g.,
DALL-E-2), achieve state-of-the-art (SOTA) performance on various compositional
T2I benchmarks, at the cost of significant computational resources. The unCLIP
stack comprises T2I prior and diffusion image decoder. The T2I prior model
alone adds a billion parameters compared to the Latent Diffusion Models, which
increases the computational and high-quality data requirements. We introduce
ECLIPSE, a novel contrastive learning method that is both parameter and
data-efficient. ECLIPSE leverages pre-trained vision-language models (e.g.,
CLIP) to distill the knowledge into the prior model. We demonstrate that the
ECLIPSE trained prior, with only 3.3% of the parameters and trained on a mere
2.8% of the data, surpasses the baseline T2I priors with an average of 71.6%
preference score under resource-limited setting. It also attains performance on
par with SOTA big models, achieving an average of 63.36% preference score in
terms of the ability to follow the text compositions. Extensive experiments on
two unCLIP diffusion image decoders, Karlo and Kandinsky, affirm that ECLIPSE
priors consistently deliver high performance while significantly reducing
resource dependency.
ECLIPSE is a parameter and data-efficient contrastive learning method for Text-to-Image models that outperforms larger models with fewer resources and maintains high preference scores.
T2I diffusion models unCLIP models DALL-E-2 T2I prior diffusion image decoder contrastive learning pre-trained vision-language models CLIP ECLIPSE Karlo Kandinsky preference score