OD-VAE: An Omni-dimensional Video Compressor for Improving Latent Video
  Diffusion Model
Variational Autoencoder (VAE), compressing videos into latent
representations, is a crucial preceding component of Latent Video Diffusion
Models (LVDMs). With the same reconstruction quality, the more sufficient the
VAE's compression for videos is, the more efficient the LVDMs are. However,
most LVDMs utilize 2D image VAE, whose compression for videos is only in the
spatial dimension and often ignored in the temporal dimension. How to conduct
temporal compression for videos in a VAE to obtain more concise latent
representations while promising accurate reconstruction is seldom explored. To
fill this gap, we propose an omni-dimension compression VAE, named OD-VAE,
which can temporally and spatially compress videos. Although OD-VAE's more
sufficient compression brings a great challenge to video reconstruction, it can
still achieve high reconstructed accuracy by our fine design. To obtain a
better trade-off between video reconstruction quality and compression speed,
four variants of OD-VAE are introduced and analyzed. In addition, a novel tail
initialization is designed to train OD-VAE more efficiently, and a novel
inference strategy is proposed to enable OD-VAE to handle videos of arbitrary
length with limited GPU memory. Comprehensive experiments on video
reconstruction and LVDM-based video generation demonstrate the effectiveness
and efficiency of our proposed methods.
66d7f0aa6776d11459adae38
Liuhan Chen
claimed_verified
2024-12-30T19:38:55.516Z
65c0588569429d85dc594a85
/avatars/822203d982519b999247a53235e30c39.svg
liuhan
user
LiuhanChen
66d7f0aa6776d11459adae39
Zongjian Li
66d7f0aa6776d11459adae3a
Bin Lin
claimed_verified
2024-12-03T11:16:54.034Z
6367a8175bb06007ea099b8f
https://cdn-avatars.huggingface.co/v1/production/uploads/6367a8175bb06007ea099b8f/IjG7HyWyWRlVt_XwRbxRW.jpeg
linbin
user
LanguageBind
66d7f0aa6776d11459adae3b
Bin Zhu
66d7f0aa6776d11459adae3c
Qian Wang
66d7f0aa6776d11459adae3d
Shenghai Yuan
claimed_verified
2024-09-06T09:06:08.240Z
63468720dd6d90d82ccf3450
https://cdn-avatars.huggingface.co/v1/production/uploads/63468720dd6d90d82ccf3450/tVBFlmZNz8FRMkOrDaDID.jpeg
YSH
user
BestWishYsh
66d7f0aa6776d11459adae3e
Xing Zhou
66d7f0aa6776d11459adae3f
Xinghua Cheng
66d7f0aa6776d11459adae40
Li Yuan
claimed_verified
2025-04-04T07:11:39.472Z
66135a5e50350afe76beebce
/avatars/370a4b83949355feb050c2cb0425c264.svg
yl2488
user
yl2488
OD-VAE: An Omni-dimensional Video Compressor for Improving Latent Video
  Diffusion Model
Variational Autoencoder (VAE), compressing videos into latent
representations, is a crucial preceding component of Latent Video Diffusion
Models (LVDMs). With the same reconstruction quality, the more sufficient the
VAE's compression for videos is, the more efficient the LVDMs are. However,
most LVDMs utilize 2D image VAE, whose compression for videos is only in the
spatial dimension and often ignored in the temporal dimension. How to conduct
temporal compression for videos in a VAE to obtain more concise latent
representations while promising accurate reconstruction is seldom explored. To
fill this gap, we propose an omni-dimension compression VAE, named OD-VAE,
which can temporally and spatially compress videos. Although OD-VAE's more
sufficient compression brings a great challenge to video reconstruction, it can
still achieve high reconstructed accuracy by our fine design. To obtain a
better trade-off between video reconstruction quality and compression speed,
four variants of OD-VAE are introduced and analyzed. In addition, a novel tail
initialization is designed to train OD-VAE more efficiently, and a novel
inference strategy is proposed to enable OD-VAE to handle videos of arbitrary
length with limited GPU memory. Comprehensive experiments on video
reconstruction and LVDM-based video generation demonstrate the effectiveness
and efficiency of our proposed methods.
A novel omni-dimensional VAE (OD-VAE) is proposed to improve video compression by integrating both spatial and temporal dimensions, enhancing the efficiency and quality of latent video diffusion models.
Variational Autoencoder Latent Video Diffusion Models omni-dimension compression video reconstruction compression speed tail initialization inference strategy latent representations