Timestep Embedding Tells: It's Time to Cache for Video Diffusion Model
As a fundamental backbone for video generation, diffusion models are
challenged by low inference speed due to the sequential nature of denoising.
Previous methods speed up the models by caching and reusing model outputs at
uniformly selected timesteps. However, such a strategy neglects the fact that
differences among model outputs are not uniform across timesteps, which hinders
selecting the appropriate model outputs to cache, leading to a poor balance
between inference efficiency and visual quality. In this study, we introduce
Timestep Embedding Aware Cache (TeaCache), a training-free caching approach
that estimates and leverages the fluctuating differences among model outputs
across timesteps. Rather than directly using the time-consuming model outputs,
TeaCache focuses on model inputs, which have a strong correlation with the
modeloutputs while incurring negligible computational cost. TeaCache first
modulates the noisy inputs using the timestep embeddings to ensure their
differences better approximating those of model outputs. TeaCache then
introduces a rescaling strategy to refine the estimated differences and
utilizes them to indicate output caching. Experiments show that TeaCache
achieves up to 4.41x acceleration over Open-Sora-Plan with negligible (-0.07%
Vbench score) degradation of visual quality.
674d371b5e924d18e56a450c
Feng Liu
claimed_verified
2024-12-02T09:34:03.626Z
661658b9e398e4f0757221ce
/avatars/9e9d55145661d00e2a9e029ba2e085d8.svg
Liu Feng
user
LiewFeng
674d371b5e924d18e56a450d
Shiwei Zhang
admin_assigned
2024-12-02T09:11:39.991Z
63a1200045edac9f7508bae9
/avatars/e84f0b045f32c5b8b4da43458650b925.svg
Shiwei Zhang
user
StevenZhang
674d371b5e924d18e56a450e
Xiaofeng Wang
claimed_verified
2024-12-02T08:51:21.743Z
6426616ea5ec4a5cbc535634
https://cdn-avatars.huggingface.co/v1/production/uploads/6426616ea5ec4a5cbc535634/hH6JsxnXeakH3mBTeNNmO.jpeg
JeffWang
user
Jeff-Wang
674d371b5e924d18e56a450f
Yujie Wei
admin_assigned
2024-12-02T09:12:00.442Z
637f70d6fab5db9101c3dfc8
https://cdn-avatars.huggingface.co/v1/production/uploads/637f70d6fab5db9101c3dfc8/NgkYNXWLDavLbrnCby2Fl.jpeg
Yujie Wei
user
weilllllls
674d371b5e924d18e56a4510
Haonan Qiu
admin_assigned
2024-12-02T09:12:05.919Z
63023b6ab002e9a4a2152890
/avatars/cae8ba0a8d61fb4e576934431f43991b.svg
Haonan Qiu
user
MoonQiu
674d371b5e924d18e56a4511
Yuzhong Zhao
674d371b5e924d18e56a4512
Yingya Zhang
claimed_verified
2024-12-02T09:34:01.699Z
66f3dd5c0fc8cc6ae16af8d5
https://cdn-avatars.huggingface.co/v1/production/uploads/66f3dd5c0fc8cc6ae16af8d5/xd-PuVvj5Ber9yvAFm3jq.jpeg
voidoc
user
ldmLDM77
674d371b5e924d18e56a4513
Qixiang Ye
674d371b5e924d18e56a4514
Fang Wan
Timestep Embedding Tells: It's Time to Cache for Video Diffusion Model
As a fundamental backbone for video generation, diffusion models are
challenged by low inference speed due to the sequential nature of denoising.
Previous methods speed up the models by caching and reusing model outputs at
uniformly selected timesteps. However, such a strategy neglects the fact that
differences among model outputs are not uniform across timesteps, which hinders
selecting the appropriate model outputs to cache, leading to a poor balance
between inference efficiency and visual quality. In this study, we introduce
Timestep Embedding Aware Cache (TeaCache), a training-free caching approach
that estimates and leverages the fluctuating differences among model outputs
across timesteps. Rather than directly using the time-consuming model outputs,
TeaCache focuses on model inputs, which have a strong correlation with the
modeloutputs while incurring negligible computational cost. TeaCache first
modulates the noisy inputs using the timestep embeddings to ensure their
differences better approximating those of model outputs. TeaCache then
introduces a rescaling strategy to refine the estimated differences and
utilizes them to indicate output caching. Experiments show that TeaCache
achieves up to 4.41x acceleration over Open-Sora-Plan with negligible (-0.07%
Vbench score) degradation of visual quality.
A new training-free caching method called TeaCache improves inference speed in diffusion models for video generation without significantly compromising visual quality.
diffusion models inference speed denoising model outputs timestep embeddings rescaling strategy TeaCache Open-Sora-Plan Vbench score