I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion
  Models
Video synthesis has recently made remarkable strides benefiting from the
rapid development of diffusion models. However, it still encounters challenges
in terms of semantic accuracy, clarity and spatio-temporal continuity. They
primarily arise from the scarcity of well-aligned text-video data and the
complex inherent structure of videos, making it difficult for the model to
simultaneously ensure semantic and qualitative excellence. In this report, we
propose a cascaded I2VGen-XL approach that enhances model performance by
decoupling these two factors and ensures the alignment of the input data by
utilizing static images as a form of crucial guidance. I2VGen-XL consists of
two stages: i) the base stage guarantees coherent semantics and preserves
content from input images by using two hierarchical encoders, and ii) the
refinement stage enhances the video's details by incorporating an additional
brief text and improves the resolution to 1280times720. To improve the
diversity, we collect around 35 million single-shot text-video pairs and 6
billion text-image pairs to optimize the model. By this means, I2VGen-XL can
simultaneously enhance the semantic accuracy, continuity of details and clarity
of generated videos. Through extensive experiments, we have investigated the
underlying principles of I2VGen-XL and compared it with current top methods,
which can demonstrate its effectiveness on diverse data. The source code and
models will be publicly available at https://i2vgen-xl.github.io.
654af0fce4432d6c7b0dfe91
Shiwei Zhang
admin_assigned
2023-11-08T08:27:25.768Z
63a1200045edac9f7508bae9
/avatars/e84f0b045f32c5b8b4da43458650b925.svg
Shiwei Zhang
user
StevenZhang
654af0fce4432d6c7b0dfe92
Jiayu Wang
654af0fce4432d6c7b0dfe93
Yingya Zhang
654af0fce4432d6c7b0dfe94
Kang Zhao
654af0fce4432d6c7b0dfe95
Hangjie Yuan
admin_assigned
2023-11-08T08:37:26.728Z
649d54b314afbb10ce2a9eeb
/avatars/15c325d8c2273ff63569f23015e98486.svg
Hangjie Yuan
user
JacobYuan
654af0fce4432d6c7b0dfe96
Zhiwu Qin
654af0fce4432d6c7b0dfe97
Xiang Wang
654af0fce4432d6c7b0dfe98
Deli Zhao
654af0fce4432d6c7b0dfe99
Jingren Zhou
admin_assigned
2023-11-08T08:38:48.065Z
602f88f5e8149a962412a667
/avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
Zhou
user
Jingren
I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion
  Models
Video synthesis has recently made remarkable strides benefiting from the
rapid development of diffusion models. However, it still encounters challenges
in terms of semantic accuracy, clarity and spatio-temporal continuity. They
primarily arise from the scarcity of well-aligned text-video data and the
complex inherent structure of videos, making it difficult for the model to
simultaneously ensure semantic and qualitative excellence. In this report, we
propose a cascaded I2VGen-XL approach that enhances model performance by
decoupling these two factors and ensures the alignment of the input data by
utilizing static images as a form of crucial guidance. I2VGen-XL consists of
two stages: i) the base stage guarantees coherent semantics and preserves
content from input images by using two hierarchical encoders, and ii) the
refinement stage enhances the video's details by incorporating an additional
brief text and improves the resolution to 1280times720. To improve the
diversity, we collect around 35 million single-shot text-video pairs and 6
billion text-image pairs to optimize the model. By this means, I2VGen-XL can
simultaneously enhance the semantic accuracy, continuity of details and clarity
of generated videos. Through extensive experiments, we have investigated the
underlying principles of I2VGen-XL and compared it with current top methods,
which can demonstrate its effectiveness on diverse data. The source code and
models will be publicly available at https://i2vgen-xl.github.io.
The cascaded I2VGen-XL approach uses hierarchical encoders and text-image pairs to improve the semantic accuracy, continuity, and clarity of generated videos.
diffusion models semantic accuracy spatio-temporal continuity I2VGen-XL hierarchical encoders text-video pairs text-image pairs