On the Scalability of Diffusion-based Text-to-Image Generation
Scaling up model and data size has been quite successful for the evolution of
LLMs. However, the scaling law for the diffusion based text-to-image (T2I)
models is not fully explored. It is also unclear how to efficiently scale the
model for better performance at reduced cost. The different training settings
and expensive training cost make a fair model comparison extremely difficult.
In this work, we empirically study the scaling properties of diffusion based
T2I models by performing extensive and rigours ablations on scaling both
denoising backbones and training set, including training scaled UNet and
Transformer variants ranging from 0.4B to 4B parameters on datasets upto 600M
images. For model scaling, we find the location and amount of cross attention
distinguishes the performance of existing UNet designs. And increasing the
transformer blocks is more parameter-efficient for improving text-image
alignment than increasing channel numbers. We then identify an efficient UNet
variant, which is 45% smaller and 28% faster than SDXL's UNet. On the data
scaling side, we show the quality and diversity of the training set matters
more than simply dataset size. Increasing caption density and diversity
improves text-image alignment performance and the learning efficiency. Finally,
we provide scaling functions to predict the text-image alignment performance as
functions of the scale of model size, compute and dataset size.
660e17b6bc5f12490fd40f46
Hao Li
claimed_verified
2024-04-05T10:05:47.329Z
633bebbb20ceb9fcae5a5def
/avatars/3b0c6587c5b9939e1da6a6dfdbe73ad7.svg
Hao
user
hlimax
660e17b6bc5f12490fd40f47
Yang Zou
660e17b6bc5f12490fd40f48
Ying Wang
660e17b6bc5f12490fd40f49
Orchid Majumder
660e17b6bc5f12490fd40f4a
Yusheng Xie
claimed_verified
2024-04-04T07:55:54.856Z
627b3008f5ef71071a3f7f51
/avatars/0e17d4f424d67a5983f2915d52c26860.svg
Yusheng Xie
user
yushx
660e17b6bc5f12490fd40f4b
R. Manmatha
660e17b6bc5f12490fd40f4c
Ashwin Swaminathan
660e17b6bc5f12490fd40f4d
Zhuowen Tu
660e17b6bc5f12490fd40f4e
Stefano Ermon
660e17b6bc5f12490fd40f4f
Stefano Soatto
On the Scalability of Diffusion-based Text-to-Image Generation
Scaling up model and data size has been quite successful for the evolution of
LLMs. However, the scaling law for the diffusion based text-to-image (T2I)
models is not fully explored. It is also unclear how to efficiently scale the
model for better performance at reduced cost. The different training settings
and expensive training cost make a fair model comparison extremely difficult.
In this work, we empirically study the scaling properties of diffusion based
T2I models by performing extensive and rigours ablations on scaling both
denoising backbones and training set, including training scaled UNet and
Transformer variants ranging from 0.4B to 4B parameters on datasets upto 600M
images. For model scaling, we find the location and amount of cross attention
distinguishes the performance of existing UNet designs. And increasing the
transformer blocks is more parameter-efficient for improving text-image
alignment than increasing channel numbers. We then identify an efficient UNet
variant, which is 45% smaller and 28% faster than SDXL's UNet. On the data
scaling side, we show the quality and diversity of the training set matters
more than simply dataset size. Increasing caption density and diversity
improves text-image alignment performance and the learning efficiency. Finally,
we provide scaling functions to predict the text-image alignment performance as
functions of the scale of model size, compute and dataset size.
Empirical study examines the scaling properties of diffusion-based text-to-image models, identifying efficient architecture and dataset practices to improve performance and reduce cost.
diffusion based T2I models denoising backbones scaling UNet Transformer variants cross attention parameter-efficient text-image alignment caption density scaling functions text-image alignment performance