VideoGen-of-Thought: A Collaborative Framework for Multi-Shot Video
  Generation
Current video generation models excel at generating short clips but still
struggle with creating multi-shot, movie-like videos. Existing models trained
on large-scale data on the back of rich computational resources are
unsurprisingly inadequate for maintaining a logical storyline and visual
consistency across multiple shots of a cohesive script since they are often
trained with a single-shot objective. To this end, we propose
VideoGen-of-Thought (VGoT), a collaborative and training-free architecture
designed specifically for multi-shot video generation. VGoT is designed with
three goals in mind as follows. Multi-Shot Video Generation: We divide the
video generation process into a structured, modular sequence, including (1)
Script Generation, which translates a curt story into detailed prompts for each
shot; (2) Keyframe Generation, responsible for creating visually consistent
keyframes faithful to character portrayals; and (3) Shot-Level Video
Generation, which transforms information from scripts and keyframes into shots;
(4) Smoothing Mechanism that ensures a consistent multi-shot output. Reasonable
Narrative Design: Inspired by cinematic scriptwriting, our prompt generation
approach spans five key domains, ensuring logical consistency, character
development, and narrative flow across the entire video. Cross-Shot
Consistency: We ensure temporal and identity consistency by leveraging
identity-preserving (IP) embeddings across shots, which are automatically
created from the narrative. Additionally, we incorporate a cross-shot smoothing
mechanism, which integrates a reset boundary that effectively combines latent
features from adjacent shots, resulting in smooth transitions and maintaining
visual coherence throughout the video. Our experiments demonstrate that VGoT
surpasses existing video generation methods in producing high-quality,
coherent, multi-shot videos.
674fcf94d367eb55390fc796
Mingzhe Zheng
claimed_verified
2024-12-05T09:18:52.776Z
62f0c4abe2999b231e5a893c
/avatars/90da268b877a7ffe6665075c84018a83.svg
Mingzhe Zheng
user
Dunge0nMaster
674fcf94d367eb55390fc797
Yongqi Xu
admin_assigned
2024-12-04T11:53:56.051Z
6614caef2da465c162b52abc
/avatars/81b71cb400c01ac10bfac9d1a4f2a9fc.svg
Yongqi Xu
user
Cheliosoops
674fcf94d367eb55390fc798
Haojian Huang
admin_assigned
2024-12-04T11:53:49.786Z
657a776153e2fa36fed33259
/avatars/f60777abe61bc6743e4cef0ede301295.svg
Haojian Huang
user
Jethro37
674fcf94d367eb55390fc799
Xuran Ma
674fcf94d367eb55390fc79a
Yexin Liu
admin_assigned
2024-12-04T11:53:28.015Z
6497ff395b5d43c1c77d2a11
/avatars/e3704701ed3ea0c1c165001521f40086.svg
Yexin Liu
user
AI4VR
674fcf94d367eb55390fc79b
Wenjie Shu
674fcf94d367eb55390fc79c
Yatian Pang
674fcf94d367eb55390fc79d
Feilong Tang
admin_assigned
2024-12-04T11:53:06.726Z
67467d9bf7b7f6fffeee8f09
/avatars/da5dd1acd79478540c32b34060710e1b.svg
tangfeilong
user
tfl01
674fcf94d367eb55390fc79e
Qifeng Chen
admin_assigned
2024-12-04T11:52:48.169Z
6467b121e7a6a374fd19b44b
/avatars/3f2874d58986d651aef55e3408b05700.svg
Qifeng Chen
user
cqf
674fcf94d367eb55390fc79f
Harry Yang
674fcf94d367eb55390fc7a0
Ser-Nam Lim
admin_assigned
2024-12-04T10:36:17.200Z
65884498ce38d143c435d279
/avatars/96a917c40680a79a37dcb8fa75014c21.svg
Ser Nam Lim
user
sernam
VideoGen-of-Thought: A Collaborative Framework for Multi-Shot Video
  Generation
Current video generation models excel at generating short clips but still
struggle with creating multi-shot, movie-like videos. Existing models trained
on large-scale data on the back of rich computational resources are
unsurprisingly inadequate for maintaining a logical storyline and visual
consistency across multiple shots of a cohesive script since they are often
trained with a single-shot objective. To this end, we propose
VideoGen-of-Thought (VGoT), a collaborative and training-free architecture
designed specifically for multi-shot video generation. VGoT is designed with
three goals in mind as follows. Multi-Shot Video Generation: We divide the
video generation process into a structured, modular sequence, including (1)
Script Generation, which translates a curt story into detailed prompts for each
shot; (2) Keyframe Generation, responsible for creating visually consistent
keyframes faithful to character portrayals; and (3) Shot-Level Video
Generation, which transforms information from scripts and keyframes into shots;
(4) Smoothing Mechanism that ensures a consistent multi-shot output. Reasonable
Narrative Design: Inspired by cinematic scriptwriting, our prompt generation
approach spans five key domains, ensuring logical consistency, character
development, and narrative flow across the entire video. Cross-Shot
Consistency: We ensure temporal and identity consistency by leveraging
identity-preserving (IP) embeddings across shots, which are automatically
created from the narrative. Additionally, we incorporate a cross-shot smoothing
mechanism, which integrates a reset boundary that effectively combines latent
features from adjacent shots, resulting in smooth transitions and maintaining
visual coherence throughout the video. Our experiments demonstrate that VGoT
surpasses existing video generation methods in producing high-quality,
coherent, multi-shot videos.
VGoT generates high-quality, coherent multi-shot videos by structuring the process into script, keyframe, and shot-level generation, using IP embeddings and smoothing mechanisms for consistency and narrative flow.
VideoGen-of-Thought VGoT multi-shot video generation script generation keyframe generation shot-level video generation smoothing mechanism identity-preserving embeddings IP embeddings cross-shot consistency