Bigger is not Always Better: Scaling Properties of Latent Diffusion
  Models
We study the scaling properties of latent diffusion models (LDMs) with an
emphasis on their sampling efficiency. While improved network architecture and
inference algorithms have shown to effectively boost sampling efficiency of
diffusion models, the role of model size -- a critical determinant of sampling
efficiency -- has not been thoroughly examined. Through empirical analysis of
established text-to-image diffusion models, we conduct an in-depth
investigation into how model size influences sampling efficiency across varying
sampling steps. Our findings unveil a surprising trend: when operating under a
given inference budget, smaller models frequently outperform their larger
equivalents in generating high-quality results. Moreover, we extend our study
to demonstrate the generalizability of the these findings by applying various
diffusion samplers, exploring diverse downstream tasks, evaluating
post-distilled models, as well as comparing performance relative to training
compute. These findings open up new pathways for the development of LDM scaling
strategies which can be employed to enhance generative capabilities within
limited inference budgets.
660cca736601973e7a2fdc98
Kangfu Mei
660cca736601973e7a2fdc99
Zhengzhong Tu
claimed_verified
2025-06-03T09:18:06.980Z
62548d5fef3debb2ddf91217
/avatars/14975b45568f9c399c92c3986b6ce83e.svg
Zhengzhong Tu
user
vztu
660cca736601973e7a2fdc9a
Mauricio Delbracio
660cca736601973e7a2fdc9b
Hossein Talebi
660cca736601973e7a2fdc9c
Vishal M. Patel
660cca736601973e7a2fdc9d
Peyman Milanfar
Bigger is not Always Better: Scaling Properties of Latent Diffusion
  Models
We study the scaling properties of latent diffusion models (LDMs) with an
emphasis on their sampling efficiency. While improved network architecture and
inference algorithms have shown to effectively boost sampling efficiency of
diffusion models, the role of model size -- a critical determinant of sampling
efficiency -- has not been thoroughly examined. Through empirical analysis of
established text-to-image diffusion models, we conduct an in-depth
investigation into how model size influences sampling efficiency across varying
sampling steps. Our findings unveil a surprising trend: when operating under a
given inference budget, smaller models frequently outperform their larger
equivalents in generating high-quality results. Moreover, we extend our study
to demonstrate the generalizability of the these findings by applying various
diffusion samplers, exploring diverse downstream tasks, evaluating
post-distilled models, as well as comparing performance relative to training
compute. These findings open up new pathways for the development of LDM scaling
strategies which can be employed to enhance generative capabilities within
limited inference budgets.
Smaller latent diffusion models often outperform larger ones in terms of sampling efficiency and generate high-quality results within limited inference budgets.
latent diffusion models sampling efficiency diffusion models model size inference budget sampling steps diffusion samplers post-distilled models generative capabilities