ControlVideo: Training-free Controllable Text-to-Video Generation
Text-driven diffusion models have unlocked unprecedented abilities in image
generation, whereas their video counterpart still lags behind due to the
excessive training cost of temporal modeling. Besides the training burden, the
generated videos also suffer from appearance inconsistency and structural
flickers, especially in long video synthesis. To address these challenges, we
design a training-free framework called ControlVideo to enable
natural and efficient text-to-video generation. ControlVideo, adapted from
ControlNet, leverages coarsely structural consistency from input motion
sequences, and introduces three modules to improve video generation. Firstly,
to ensure appearance coherence between frames, ControlVideo adds fully
cross-frame interaction in self-attention modules. Secondly, to mitigate the
flicker effect, it introduces an interleaved-frame smoother that employs frame
interpolation on alternated frames. Finally, to produce long videos
efficiently, it utilizes a hierarchical sampler that separately synthesizes
each short clip with holistic coherency. Empowered with these modules,
ControlVideo outperforms the state-of-the-arts on extensive motion-prompt pairs
quantitatively and qualitatively. Notably, thanks to the efficient designs, it
generates both short and long videos within several minutes using one NVIDIA
2080Ti. Code is available at https://github.com/YBYBZhang/ControlVideo.
646c140131968a60a022c3c4
Yabo Zhang
claimed_verified
2023-05-24T07:51:28.713Z
62cd752ba9be5c19555c2b4c
/avatars/ed72684f0f2139516ccde24cd467cea6.svg
YaboZhang
user
Yabo
646c140131968a60a022c3c5
Yuxiang Wei
claimed_verified
2024-01-08T08:49:53.109Z
6308bef17dc1b1a54cc9ff56
/avatars/d3793f6192a7441a575e27996c065030.svg
Yuxiang Wei
user
csyxwei
646c140131968a60a022c3c6
Dongsheng Jiang
646c140131968a60a022c3c7
Xiaopeng Zhang
646c140131968a60a022c3c8
Wangmeng Zuo
646c140131968a60a022c3c9
Qi Tian
ControlVideo: Training-free Controllable Text-to-Video Generation
Text-driven diffusion models have unlocked unprecedented abilities in image
generation, whereas their video counterpart still lags behind due to the
excessive training cost of temporal modeling. Besides the training burden, the
generated videos also suffer from appearance inconsistency and structural
flickers, especially in long video synthesis. To address these challenges, we
design a training-free framework called ControlVideo to enable
natural and efficient text-to-video generation. ControlVideo, adapted from
ControlNet, leverages coarsely structural consistency from input motion
sequences, and introduces three modules to improve video generation. Firstly,
to ensure appearance coherence between frames, ControlVideo adds fully
cross-frame interaction in self-attention modules. Secondly, to mitigate the
flicker effect, it introduces an interleaved-frame smoother that employs frame
interpolation on alternated frames. Finally, to produce long videos
efficiently, it utilizes a hierarchical sampler that separately synthesizes
each short clip with holistic coherency. Empowered with these modules,
ControlVideo outperforms the state-of-the-arts on extensive motion-prompt pairs
quantitatively and qualitatively. Notably, thanks to the efficient designs, it
generates both short and long videos within several minutes using one NVIDIA
2080Ti. Code is available at https://github.com/YBYBZhang/ControlVideo.
ControlVideo, a training-free framework adapted from ControlNet, improves text-to-video generation by ensuring appearance coherence, reducing flicker, and efficiently producing long videos.
text-driven diffusion models video generation training-free framework ControlVideo ControlNet fully cross-frame interaction self-attention modules interleaved-frame smoother frame interpolation hierarchical sampler