SnapFusion: Text-to-Image Diffusion Model on Mobile Devices within Two
  Seconds
Text-to-image diffusion models can create stunning images from natural
language descriptions that rival the work of professional artists and
photographers. However, these models are large, with complex network
architectures and tens of denoising iterations, making them computationally
expensive and slow to run. As a result, high-end GPUs and cloud-based inference
are required to run diffusion models at scale. This is costly and has privacy
implications, especially when user data is sent to a third party. To overcome
these challenges, we present a generic approach that, for the first time,
unlocks running text-to-image diffusion models on mobile devices in less than
2 seconds. We achieve so by introducing efficient network architecture and
improving step distillation. Specifically, we propose an efficient UNet by
identifying the redundancy of the original model and reducing the computation
of the image decoder via data distillation. Further, we enhance the step
distillation by exploring training strategies and introducing regularization
from classifier-free guidance. Our extensive experiments on MS-COCO show that
our model with 8 denoising steps achieves better FID and CLIP scores than
Stable Diffusion v1.5 with 50 steps. Our work democratizes content creation
by bringing powerful text-to-image diffusion models to the hands of users.
64797f1b12ae749b62ee7671
Yanyu Li
64797f1b12ae749b62ee7672
Huan Wang
claimed_verified
2023-06-02T12:46:32.679Z
62b624f3b52bef716e248fd7
https://cdn-avatars.huggingface.co/v1/production/uploads/62b624f3b52bef716e248fd7/AllcccKH-eBWduA8KVnOQ.png
Huan Wang
user
Huan-WhoRegisteredMyName
64797f1b12ae749b62ee7673
Qing Jin
64797f1b12ae749b62ee7674
Ju Hu
64797f1b12ae749b62ee7675
Pavlo Chemerys
64797f1b12ae749b62ee7676
Yun Fu
64797f1b12ae749b62ee7677
Yanzhi Wang
64797f1b12ae749b62ee7678
Sergey Tulyakov
64797f1b12ae749b62ee7679
Jian Ren
claimed_verified
2023-06-05T10:08:00.227Z
61f19829233c91cbd2f79e70
/avatars/a0735a94542b4f7cda5aed8bc4be0538.svg
Jian Ren
user
alanspike
SnapFusion: Text-to-Image Diffusion Model on Mobile Devices within Two
  Seconds
Text-to-image diffusion models can create stunning images from natural
language descriptions that rival the work of professional artists and
photographers. However, these models are large, with complex network
architectures and tens of denoising iterations, making them computationally
expensive and slow to run. As a result, high-end GPUs and cloud-based inference
are required to run diffusion models at scale. This is costly and has privacy
implications, especially when user data is sent to a third party. To overcome
these challenges, we present a generic approach that, for the first time,
unlocks running text-to-image diffusion models on mobile devices in less than
2 seconds. We achieve so by introducing efficient network architecture and
improving step distillation. Specifically, we propose an efficient UNet by
identifying the redundancy of the original model and reducing the computation
of the image decoder via data distillation. Further, we enhance the step
distillation by exploring training strategies and introducing regularization
from classifier-free guidance. Our extensive experiments on MS-COCO show that
our model with 8 denoising steps achieves better FID and CLIP scores than
Stable Diffusion v1.5 with 50 steps. Our work democratizes content creation
by bringing powerful text-to-image diffusion models to the hands of users.
Text-to-image diffusion models are optimized for mobile devices through efficient architecture and improved distillation techniques, achieving high-quality image generation with reduced computational requirements.
diffusion models text-to-image UNet denoising steps FID CLIP scores classifier-free guidance data distillation step distillation MS-COCO