Faster Diffusion: Rethinking the Role of UNet Encoder in Diffusion
  Models
One of the key components within diffusion models is the UNet for noise
prediction. While several works have explored basic properties of the UNet
decoder, its encoder largely remains unexplored. In this work, we conduct the
first comprehensive study of the UNet encoder. We empirically analyze the
encoder features and provide insights to important questions regarding their
changes at the inference process. In particular, we find that encoder features
change gently, whereas the decoder features exhibit substantial variations
across different time-steps. This finding inspired us to omit the encoder at
certain adjacent time-steps and reuse cyclically the encoder features in the
previous time-steps for the decoder. Further based on this observation, we
introduce a simple yet effective encoder propagation scheme to accelerate the
diffusion sampling for a diverse set of tasks. By benefiting from our
propagation scheme, we are able to perform in parallel the decoder at certain
adjacent time-steps. Additionally, we introduce a prior noise injection method
to improve the texture details in the generated image. Besides the standard
text-to-image task, we also validate our approach on other tasks:
text-to-video, personalized generation and reference-guided generation. Without
utilizing any knowledge distillation technique, our approach accelerates both
the Stable Diffusion (SD) and the DeepFloyd-IF models sampling by 41% and
24% respectively, while maintaining high-quality generation performance. Our
code is available in
https://github.com/hutaiHang/Faster-Diffusion{FasterDiffusion}.
657fd43a69cfe9b4cd736b64
Senmao Li
claimed_verified
2023-12-18T14:30:43.064Z
637e1cf4f09bf2498c543a73
/avatars/3c0e4e15602a384839bee0c23029f58a.svg
Senmao Li
user
senmaonk
657fd43a69cfe9b4cd736b65
Taihang Hu
admin_assigned
2023-12-18T11:10:03.301Z
634bec6ca5d4e109c6aa4895
/avatars/2f55327097b3782f6f4f59041f3b04b5.svg
hu
user
taihang
657fd43a69cfe9b4cd736b66
Fahad Shahbaz Khan
657fd43a69cfe9b4cd736b67
Linxuan Li
657fd43a69cfe9b4cd736b68
Shiqi Yang
admin_assigned
2023-12-18T11:10:36.590Z
648ea2a0df53671f33ee01ed
/avatars/b0a6af7a86f442a710c694a751936922.svg
Shiqi Yang
user
yangsq
657fd43a69cfe9b4cd736b69
Yaxing Wang
admin_assigned
2023-12-18T11:10:43.809Z
633242c16eae0bb0a01ea9d7
/avatars/d0c6ac176332a15238eff84cb8832a00.svg
Yaxing Wang
user
Yaxing
657fd43a69cfe9b4cd736b6a
Ming-Ming Cheng
admin_assigned
2023-12-18T11:11:03.184Z
64e496ae0195913c7fa91c66
/avatars/23f274f0a3b4ef6ded35205df9bfb564.svg
chengmingming
user
mingming8688
657fd43a69cfe9b4cd736b6b
Jian Yang
Faster Diffusion: Rethinking the Role of UNet Encoder in Diffusion
  Models
One of the key components within diffusion models is the UNet for noise
prediction. While several works have explored basic properties of the UNet
decoder, its encoder largely remains unexplored. In this work, we conduct the
first comprehensive study of the UNet encoder. We empirically analyze the
encoder features and provide insights to important questions regarding their
changes at the inference process. In particular, we find that encoder features
change gently, whereas the decoder features exhibit substantial variations
across different time-steps. This finding inspired us to omit the encoder at
certain adjacent time-steps and reuse cyclically the encoder features in the
previous time-steps for the decoder. Further based on this observation, we
introduce a simple yet effective encoder propagation scheme to accelerate the
diffusion sampling for a diverse set of tasks. By benefiting from our
propagation scheme, we are able to perform in parallel the decoder at certain
adjacent time-steps. Additionally, we introduce a prior noise injection method
to improve the texture details in the generated image. Besides the standard
text-to-image task, we also validate our approach on other tasks:
text-to-video, personalized generation and reference-guided generation. Without
utilizing any knowledge distillation technique, our approach accelerates both
the Stable Diffusion (SD) and the DeepFloyd-IF models sampling by 41% and
24% respectively, while maintaining high-quality generation performance. Our
code is available in
https://github.com/hutaiHang/Faster-Diffusion{FasterDiffusion}.
Introducing an encoder propagation scheme accelerates diffusion sampling without distilling knowledge, yielding faster and high-quality image and video generation.
diffusion models UNet encoder decoder noise prediction noise injection diffusion sampling Stable Diffusion DeepFloyd-IF prior noise injection text-to-image text-to-video personalized generation reference-guided generation