SEINE: Short-to-Long Video Diffusion Model for Generative Transition and
  Prediction
Recently video generation has achieved substantial progress with realistic
results. Nevertheless, existing AI-generated videos are usually very short
clips ("shot-level") depicting a single scene. To deliver a coherent long video
("story-level"), it is desirable to have creative transition and prediction
effects across different clips. This paper presents a short-to-long video
diffusion model, SEINE, that focuses on generative transition and prediction.
The goal is to generate high-quality long videos with smooth and creative
transitions between scenes and varying lengths of shot-level videos.
Specifically, we propose a random-mask video diffusion model to automatically
generate transitions based on textual descriptions. By providing the images of
different scenes as inputs, combined with text-based control, our model
generates transition videos that ensure coherence and visual quality.
Furthermore, the model can be readily extended to various tasks such as
image-to-video animation and autoregressive video prediction. To conduct a
comprehensive evaluation of this new generative task, we propose three
assessing criteria for smooth and creative transition: temporal consistency,
semantic similarity, and video-text semantic alignment. Extensive experiments
validate the effectiveness of our approach over existing methods for generative
transition and prediction, enabling the creation of story-level long videos.
Project page: https://vchitect.github.io/SEINE-project/ .
6541ca5d5242a5242f4a9852
Xinyuan Chen
6541ca5d5242a5242f4a9853
Yaohui Wang
6541ca5d5242a5242f4a9854
Lingjun Zhang
6541ca5d5242a5242f4a9855
Shaobin Zhuang
6541ca5d5242a5242f4a9856
Xin Ma
claimed_verified
2024-09-03T07:50:42.331Z
64a51faf9ec0167a99a4d114
https://cdn-avatars.huggingface.co/v1/production/uploads/64a51faf9ec0167a99a4d114/IlJS0mwkF_ZIebaQrbJIl.jpeg
Xin Ma
user
maxin-cn
6541ca5d5242a5242f4a9857
Jiashuo Yu
claimed_verified
2024-07-29T07:10:26.377Z
64b89a14cf14c2fabe96664c
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/tEd3fBjMcEubF4plqzcUz.jpeg
Jiashuo Yu
user
awojustin
6541ca5d5242a5242f4a9858
Yali Wang
6541ca5d5242a5242f4a9859
Dahua Lin
6541ca5d5242a5242f4a985a
Yu Qiao
6541ca5d5242a5242f4a985b
Ziwei Liu
SEINE: Short-to-Long Video Diffusion Model for Generative Transition and
  Prediction
Recently video generation has achieved substantial progress with realistic
results. Nevertheless, existing AI-generated videos are usually very short
clips ("shot-level") depicting a single scene. To deliver a coherent long video
("story-level"), it is desirable to have creative transition and prediction
effects across different clips. This paper presents a short-to-long video
diffusion model, SEINE, that focuses on generative transition and prediction.
The goal is to generate high-quality long videos with smooth and creative
transitions between scenes and varying lengths of shot-level videos.
Specifically, we propose a random-mask video diffusion model to automatically
generate transitions based on textual descriptions. By providing the images of
different scenes as inputs, combined with text-based control, our model
generates transition videos that ensure coherence and visual quality.
Furthermore, the model can be readily extended to various tasks such as
image-to-video animation and autoregressive video prediction. To conduct a
comprehensive evaluation of this new generative task, we propose three
assessing criteria for smooth and creative transition: temporal consistency,
semantic similarity, and video-text semantic alignment. Extensive experiments
validate the effectiveness of our approach over existing methods for generative
transition and prediction, enabling the creation of story-level long videos.
Project page: https://vchitect.github.io/SEINE-project/ .
A short-to-long video diffusion model, SEINE, generates smooth and creative transitions between scenes with text-based control, enabling the creation of coherent long videos.
video diffusion model SEINE generative transition prediction random-mask temporal consistency semantic similarity video-text semantic alignment image-to-video animation autoregressive video prediction