On Architectural Compression of Text-to-Image Diffusion Models
Exceptional text-to-image (T2I) generation results of Stable Diffusion models
(SDMs) come with substantial computational demands. To resolve this issue,
recent research on efficient SDMs has prioritized reducing the number of
sampling steps and utilizing network quantization. Orthogonal to these
directions, this study highlights the power of classical architectural
compression for general-purpose T2I synthesis by introducing block-removed
knowledge-distilled SDMs (BK-SDMs). We eliminate several residual and attention
blocks from the U-Net of SDMs, obtaining over a 30% reduction in the number of
parameters, MACs per sampling step, and latency. We conduct distillation-based
pretraining with only 0.22M LAION pairs (fewer than 0.1% of the full training
pairs) on a single A100 GPU. Despite being trained with limited resources, our
compact models can imitate the original SDM by benefiting from transferred
knowledge and achieve competitive results against larger multi-billion
parameter models on the zero-shot MS-COCO benchmark. Moreover, we demonstrate
the applicability of our lightweight pretrained models in personalized
generation with DreamBooth finetuning.
64701658850a938d6c56653c
Bo-Kyeong Kim
claimed_verified
2023-05-26T13:57:46.473Z
64374f10cd93f4c9a34c1239
https://cdn-avatars.huggingface.co/v1/production/uploads/64374f10cd93f4c9a34c1239/tdkdcgl18rKXLvyTEhfnG.jpeg
Bo-Kyeong Kim
user
bokyeong1015
64701658850a938d6c56653d
Hyoung-Kyu Song
claimed_verified
2023-05-26T08:07:18.582Z
61f44bab7eba274ea80b74ce
https://cdn-avatars.huggingface.co/v1/production/uploads/61f44bab7eba274ea80b74ce/BRbKX1jephdZ7D44FATl4.jpeg
Hyoung-Kyu Song
user
deepkyu
64701658850a938d6c56653e
Thibault Castells
claimed_verified
2023-05-26T13:57:50.370Z
64709c9390482b0e0f670e80
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/8hZF_LZQ6ejoS_u_knXFV.png
Thibault Castells
user
thibcastells
64701658850a938d6c56653f
Shinkook Choi
claimed_verified
2023-05-29T08:43:19.101Z
642f740f83e266575cf1f61d
https://cdn-avatars.huggingface.co/v1/production/uploads/642f740f83e266575cf1f61d/230E5P-CR7DFpCchGgnnw.jpeg
shinkookchoi
user
skchoi
On Architectural Compression of Text-to-Image Diffusion Models
Exceptional text-to-image (T2I) generation results of Stable Diffusion models
(SDMs) come with substantial computational demands. To resolve this issue,
recent research on efficient SDMs has prioritized reducing the number of
sampling steps and utilizing network quantization. Orthogonal to these
directions, this study highlights the power of classical architectural
compression for general-purpose T2I synthesis by introducing block-removed
knowledge-distilled SDMs (BK-SDMs). We eliminate several residual and attention
blocks from the U-Net of SDMs, obtaining over a 30% reduction in the number of
parameters, MACs per sampling step, and latency. We conduct distillation-based
pretraining with only 0.22M LAION pairs (fewer than 0.1% of the full training
pairs) on a single A100 GPU. Despite being trained with limited resources, our
compact models can imitate the original SDM by benefiting from transferred
knowledge and achieve competitive results against larger multi-billion
parameter models on the zero-shot MS-COCO benchmark. Moreover, we demonstrate
the applicability of our lightweight pretrained models in personalized
generation with DreamBooth finetuning.
Classical architectural compression and knowledge distillation reduce the size and computational cost of Stable Diffusion models while maintaining competitive text-to-image synthesis performance.
text-to-image (T2I) generation Stable Diffusion models (SDMs) sampling steps network quantization block-removed knowledge-distilled SDMs (BK-SDMs) residual blocks attention blocks U-Net distillation-based pretraining LAION pairs zero-shot MS-COCO benchmark DreamBooth finetuning