VEnhancer: Generative Space-Time Enhancement for Video Generation
We present VEnhancer, a generative space-time enhancement framework that
improves the existing text-to-video results by adding more details in spatial
domain and synthetic detailed motion in temporal domain. Given a generated
low-quality video, our approach can increase its spatial and temporal
resolution simultaneously with arbitrary up-sampling space and time scales
through a unified video diffusion model. Furthermore, VEnhancer effectively
removes generated spatial artifacts and temporal flickering of generated
videos. To achieve this, basing on a pretrained video diffusion model, we train
a video ControlNet and inject it to the diffusion model as a condition on low
frame-rate and low-resolution videos. To effectively train this video
ControlNet, we design space-time data augmentation as well as video-aware
conditioning. Benefiting from the above designs, VEnhancer yields to be stable
during training and shares an elegant end-to-end training manner. Extensive
experiments show that VEnhancer surpasses existing state-of-the-art video
super-resolution and space-time super-resolution methods in enhancing
AI-generated videos. Moreover, with VEnhancer, exisiting open-source
state-of-the-art text-to-video method, VideoCrafter-2, reaches the top one in
video generation benchmark -- VBench.
668f46fdd3b02463a73bc21c
Jingwen He
claimed_verified
2024-09-11T15:01:38.918Z
66a49f9b2b460286b05646b8
/avatars/5123cf7c3d00b5bc47a86011afd6abe8.svg
jwhe
user
jwhejwhe
668f46fdd3b02463a73bc21d
Tianfan Xue
668f46fdd3b02463a73bc21e
Dongyang Liu
admin_assigned
2024-08-06T15:24:51.297Z
646f1bef075e11ca78da3bb7
https://cdn-avatars.huggingface.co/v1/production/uploads/646f1bef075e11ca78da3bb7/gNS-ikyZXYeMrf4a7HTQE.jpeg
Dongyang Liu (Chris Liu)
user
Cxxs
668f46fdd3b02463a73bc21f
Xinqi Lin
admin_assigned
2024-07-11T10:35:02.938Z
644b221ef434a6a63b1a8f04
/avatars/34d106f0fd536cc7fbeb3abb94b5d775.svg
qilin
user
xinqilin
668f46fdd3b02463a73bc220
Peng Gao
668f46fdd3b02463a73bc221
Dahua Lin
admin_assigned
2024-07-11T10:35:28.492Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
668f46fdd3b02463a73bc222
Yu Qiao
668f46fdd3b02463a73bc223
Wanli Ouyang
668f46fdd3b02463a73bc224
Ziwei Liu
admin_assigned
2024-07-11T10:34:28.342Z
62ab1ac1d48b4d8b048a3473
https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
Ziwei Liu
user
liuziwei7
VEnhancer: Generative Space-Time Enhancement for Video Generation
We present VEnhancer, a generative space-time enhancement framework that
improves the existing text-to-video results by adding more details in spatial
domain and synthetic detailed motion in temporal domain. Given a generated
low-quality video, our approach can increase its spatial and temporal
resolution simultaneously with arbitrary up-sampling space and time scales
through a unified video diffusion model. Furthermore, VEnhancer effectively
removes generated spatial artifacts and temporal flickering of generated
videos. To achieve this, basing on a pretrained video diffusion model, we train
a video ControlNet and inject it to the diffusion model as a condition on low
frame-rate and low-resolution videos. To effectively train this video
ControlNet, we design space-time data augmentation as well as video-aware
conditioning. Benefiting from the above designs, VEnhancer yields to be stable
during training and shares an elegant end-to-end training manner. Extensive
experiments show that VEnhancer surpasses existing state-of-the-art video
super-resolution and space-time super-resolution methods in enhancing
AI-generated videos. Moreover, with VEnhancer, exisiting open-source
state-of-the-art text-to-video method, VideoCrafter-2, reaches the top one in
video generation benchmark -- VBench.
VEnhancer enhances video quality by increasing spatial and temporal resolution using a unified video diffusion model and video ControlNet with space-time data augmentation and video-aware conditioning.
generative space-time enhancement video diffusion model video ControlNet space-time data augmentation video-aware conditioning text-to-video video super-resolution space-time super-resolution VBench