VideoGen: A Reference-Guided Latent Diffusion Approach for High
  Definition Text-to-Video Generation
In this paper, we present VideoGen, a text-to-video generation approach,
which can generate a high-definition video with high frame fidelity and strong
temporal consistency using reference-guided latent diffusion. We leverage an
off-the-shelf text-to-image generation model, e.g., Stable Diffusion, to
generate an image with high content quality from the text prompt, as a
reference image to guide video generation. Then, we introduce an efficient
cascaded latent diffusion module conditioned on both the reference image and
the text prompt, for generating latent video representations, followed by a
flow-based temporal upsampling step to improve the temporal resolution.
Finally, we map latent video representations into a high-definition video
through an enhanced video decoder. During training, we use the first frame of a
ground-truth video as the reference image for training the cascaded latent
diffusion module. The main characterises of our approach include: the reference
image generated by the text-to-image model improves the visual fidelity; using
it as the condition makes the diffusion model focus more on learning the video
dynamics; and the video decoder is trained over unlabeled video data, thus
benefiting from high-quality easily-available videos. VideoGen sets a new
state-of-the-art in text-to-video generation in terms of both qualitative and
quantitative evaluation.
64f52ebb4dc8a69ca0155a6e
Xin Li
64f52ebb4dc8a69ca0155a6f
Wenqing Chu
admin_assigned
2023-09-04T15:05:35.638Z
634e084dc1ce28f1de916787
/avatars/f233256af736bbc7d4d7e92df4b04e98.svg
wenqing chu
user
wenqingchu
64f52ebb4dc8a69ca0155a70
Ye Wu
64f52ebb4dc8a69ca0155a71
Weihang Yuan
admin_assigned
2023-09-04T08:56:47.360Z
64c22c5dd1ca220e30433db4
/avatars/7b0f3d87ea54686b88fa37fe0e542e29.svg
yuan
user
weihangY
64f52ebb4dc8a69ca0155a72
Fanglong Liu
64f52ebb4dc8a69ca0155a73
Qi Zhang
64f52ebb4dc8a69ca0155a74
Fu Li
64f52ebb4dc8a69ca0155a75
Haocheng Feng
64f52ebb4dc8a69ca0155a76
Errui Ding
64f52ebb4dc8a69ca0155a77
Jingdong Wang
VideoGen: A Reference-Guided Latent Diffusion Approach for High
  Definition Text-to-Video Generation
In this paper, we present VideoGen, a text-to-video generation approach,
which can generate a high-definition video with high frame fidelity and strong
temporal consistency using reference-guided latent diffusion. We leverage an
off-the-shelf text-to-image generation model, e.g., Stable Diffusion, to
generate an image with high content quality from the text prompt, as a
reference image to guide video generation. Then, we introduce an efficient
cascaded latent diffusion module conditioned on both the reference image and
the text prompt, for generating latent video representations, followed by a
flow-based temporal upsampling step to improve the temporal resolution.
Finally, we map latent video representations into a high-definition video
through an enhanced video decoder. During training, we use the first frame of a
ground-truth video as the reference image for training the cascaded latent
diffusion module. The main characterises of our approach include: the reference
image generated by the text-to-image model improves the visual fidelity; using
it as the condition makes the diffusion model focus more on learning the video
dynamics; and the video decoder is trained over unlabeled video data, thus
benefiting from high-quality easily-available videos. VideoGen sets a new
state-of-the-art in text-to-video generation in terms of both qualitative and
quantitative evaluation.
VideoGen uses text-to-image models and latent diffusion to generate high-definition, temporally consistent videos from text prompts, achieving state-of-the-art performance.
text-to-video generation VideoGen latent diffusion text-to-image generation reference-guided latent diffusion cascaded latent diffusion temporal upsampling video decoder video dynamics