Control-A-Video: Controllable Text-to-Video Generation with Diffusion
  Models
This paper presents a controllable text-to-video (T2V) diffusion model, named
Video-ControlNet, that generates videos conditioned on a sequence of control
signals, such as edge or depth maps. Video-ControlNet is built on a pre-trained
conditional text-to-image (T2I) diffusion model by incorporating a
spatial-temporal self-attention mechanism and trainable temporal layers for
efficient cross-frame modeling. A first-frame conditioning strategy is proposed
to facilitate the model to generate videos transferred from the image domain as
well as arbitrary-length videos in an auto-regressive manner. Moreover,
Video-ControlNet employs a novel residual-based noise initialization strategy
to introduce motion prior from an input video, producing more coherent videos.
With the proposed architecture and strategies, Video-ControlNet can achieve
resource-efficient convergence and generate superior quality and consistent
videos with fine-grained control. Extensive experiments demonstrate its success
in various video generative tasks such as video editing and video style
transfer, outperforming previous methods in terms of consistency and quality.
Project Page: https://controlavideo.github.io/
646d6cf8fe25e5f8d9760975
Weifeng Chen
admin_assigned
2023-05-24T07:03:49.036Z
62a7fc03ac97233f161c2433
https://cdn-avatars.huggingface.co/v1/production/uploads/62a7fc03ac97233f161c2433/19wL1aCwyR-ik2WDXvns9.jpeg
weifeng
user
weifeng-chen
646d6cf8fe25e5f8d9760976
Jie Wu
claimed_verified
2023-12-22T07:37:46.613Z
6381c5d63680a7cf34e08ca9
/avatars/731467e2d80d0ae163c4a00a9e3ff9e5.svg
wujie10558@gmail.com
user
wujie10
646d6cf8fe25e5f8d9760977
Pan Xie
646d6cf8fe25e5f8d9760978
Hefeng Wu
646d6cf8fe25e5f8d9760979
Jiashi Li
646d6cf8fe25e5f8d976097a
Xin Xia
claimed_verified
2024-04-09T18:39:02.320Z
63089a78ff78e2aead8d10e7
/avatars/f326fd08abee7e31599a78923be30003.svg
XinXia
user
XiaXin-Aloys
646d6cf8fe25e5f8d976097b
Xuefeng Xiao
admin_assigned
2023-05-26T08:17:48.228Z
646b7f71df2609a541c1ab9f
/avatars/48b82e5fd9b06f41ff825507c36816cd.svg
Xuefeng Xiao
user
xiaoxuefeng
646d6cf8fe25e5f8d976097c
Liang Lin
Control-A-Video: Controllable Text-to-Video Generation with Diffusion
  Models
This paper presents a controllable text-to-video (T2V) diffusion model, named
Video-ControlNet, that generates videos conditioned on a sequence of control
signals, such as edge or depth maps. Video-ControlNet is built on a pre-trained
conditional text-to-image (T2I) diffusion model by incorporating a
spatial-temporal self-attention mechanism and trainable temporal layers for
efficient cross-frame modeling. A first-frame conditioning strategy is proposed
to facilitate the model to generate videos transferred from the image domain as
well as arbitrary-length videos in an auto-regressive manner. Moreover,
Video-ControlNet employs a novel residual-based noise initialization strategy
to introduce motion prior from an input video, producing more coherent videos.
With the proposed architecture and strategies, Video-ControlNet can achieve
resource-efficient convergence and generate superior quality and consistent
videos with fine-grained control. Extensive experiments demonstrate its success
in various video generative tasks such as video editing and video style
transfer, outperforming previous methods in terms of consistency and quality.
Project Page: https://controlavideo.github.io/
Video-ControlNet is a controllable T2V diffusion model that uses spatial-temporal self-attention and residual-based noise initialization to generate high-quality, consistent videos conditioned on control signals.
text-to-video Video-ControlNet control signals pre-trained conditional text-to-image spatial-temporal self-attention temporal layers first-frame conditioning residual-based noise initialization video editing video style transfer