AsyncDiff: Parallelizing Diffusion Models by Asynchronous Denoising
Diffusion models have garnered significant interest from the community for
their great generative ability across various applications. However, their
typical multi-step sequential-denoising nature gives rise to high cumulative
latency, thereby precluding the possibilities of parallel computation. To
address this, we introduce AsyncDiff, a universal and plug-and-play
acceleration scheme that enables model parallelism across multiple devices. Our
approach divides the cumbersome noise prediction model into multiple
components, assigning each to a different device. To break the dependency chain
between these components, it transforms the conventional sequential denoising
into an asynchronous process by exploiting the high similarity between hidden
states in consecutive diffusion steps. Consequently, each component is
facilitated to compute in parallel on separate devices. The proposed strategy
significantly reduces inference latency while minimally impacting the
generative quality. Specifically, for the Stable Diffusion v2.1, AsyncDiff
achieves a 2.7x speedup with negligible degradation and a 4.0x speedup with
only a slight reduction of 0.38 in CLIP Score, on four NVIDIA A5000 GPUs. Our
experiments also demonstrate that AsyncDiff can be readily applied to video
diffusion models with encouraging performances. The code is available at
https://github.com/czg1225/AsyncDiff.
666915f90a19d52d441b7730
Zigeng Chen
admin_assigned
2024-06-12T09:31:30.267Z
65811eeaa2284a018e51f1ba
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/dH8UZj6Kk5HJkI1DItCNm.jpeg
Zigeng Chen
user
Zigeng
666915f90a19d52d441b7731
Xinyin Ma
admin_assigned
2024-06-12T09:31:36.686Z
64396ebc21221ac7411852b3
https://cdn-avatars.huggingface.co/v1/production/uploads/64396ebc21221ac7411852b3/SR0dC8N0bdj9tZFxYPpSf.jpeg
Xinyin Ma
user
horseee
666915f90a19d52d441b7732
Gongfan Fang
admin_assigned
2024-06-12T09:31:43.353Z
646a1939c37ca1e12308fe81
/avatars/752e9d86018e7d33ad8bcd741203fd86.svg
Gongfan Fang
user
Vinnnf
666915f90a19d52d441b7733
Zhenxiong Tan
admin_assigned
2024-06-12T09:31:52.919Z
664ef197d3f8759dddfbc765
/avatars/fd3f8a844b391be6c1ac2ea89b228f5a.svg
Tan Zhen Xiong
user
ZhenXiong
666915f90a19d52d441b7734
Xinchao Wang
admin_assigned
2024-06-12T09:32:05.522Z
63fc03a50aab060792ffef39
/avatars/9d5b1bb2a41928e08176b703935133ab.svg
Wangxinchao
user
wxcTest
AsyncDiff: Parallelizing Diffusion Models by Asynchronous Denoising
Diffusion models have garnered significant interest from the community for
their great generative ability across various applications. However, their
typical multi-step sequential-denoising nature gives rise to high cumulative
latency, thereby precluding the possibilities of parallel computation. To
address this, we introduce AsyncDiff, a universal and plug-and-play
acceleration scheme that enables model parallelism across multiple devices. Our
approach divides the cumbersome noise prediction model into multiple
components, assigning each to a different device. To break the dependency chain
between these components, it transforms the conventional sequential denoising
into an asynchronous process by exploiting the high similarity between hidden
states in consecutive diffusion steps. Consequently, each component is
facilitated to compute in parallel on separate devices. The proposed strategy
significantly reduces inference latency while minimally impacting the
generative quality. Specifically, for the Stable Diffusion v2.1, AsyncDiff
achieves a 2.7x speedup with negligible degradation and a 4.0x speedup with
only a slight reduction of 0.38 in CLIP Score, on four NVIDIA A5000 GPUs. Our
experiments also demonstrate that AsyncDiff can be readily applied to video
diffusion models with encouraging performances. The code is available at
https://github.com/czg1225/AsyncDiff.
AsyncDiff, a parallel computation scheme for diffusion models, reduces inference latency without significantly impacting generative quality, demonstrating speedups on image and video tasks.
diffusion models sequential denoising model parallelism noise prediction model asynchronous process hidden states Stable Diffusion CLIP Score video diffusion models