Make-Your-Video: Customized Video Generation Using Textual and
  Structural Guidance
Creating a vivid video from the event or scenario in our imagination is a
truly fascinating experience. Recent advancements in text-to-video synthesis
have unveiled the potential to achieve this with prompts only. While text is
convenient in conveying the overall scene context, it may be insufficient to
control precisely. In this paper, we explore customized video generation by
utilizing text as context description and motion structure (e.g. frame-wise
depth) as concrete guidance. Our method, dubbed Make-Your-Video, involves
joint-conditional video generation using a Latent Diffusion Model that is
pre-trained for still image synthesis and then promoted for video generation
with the introduction of temporal modules. This two-stage learning scheme not
only reduces the computing resources required, but also improves the
performance by transferring the rich concepts available in image datasets
solely into video generation. Moreover, we use a simple yet effective causal
attention mask strategy to enable longer video synthesis, which mitigates the
potential quality degradation effectively. Experimental results show the
superiority of our method over existing baselines, particularly in terms of
temporal coherence and fidelity to users' guidance. In addition, our model
enables several intriguing applications that demonstrate potential for
practical usage.
64794f32a68454566355d7ce
Jinbo Xing
claimed_verified
2023-06-02T12:46:24.334Z
64770e86d7cf39f2e937ae9a
https://cdn-avatars.huggingface.co/v1/production/uploads/64770e86d7cf39f2e937ae9a/pLqGg2z1KzQxCGpMwds-9.jpeg
Jinbo Xing
user
Doubiiu
64794f32a68454566355d7cf
Menghan Xia
admin_assigned
2023-06-05T11:03:28.227Z
63401c89f81b9d101361f712
https://cdn-avatars.huggingface.co/v1/production/uploads/1665146415483-63401c89f81b9d101361f712.png
Richard
user
menghanxia
64794f32a68454566355d7d0
Yuxin Liu
64794f32a68454566355d7d1
Yuechen Zhang
admin_assigned
2023-06-05T11:57:52.969Z
6418554a0956be7233a1023e
https://cdn-avatars.huggingface.co/v1/production/uploads/6418554a0956be7233a1023e/9EKN0GoOpcDbvBDmAQEJf.png
zhang yuechen
user
julianjuaner
64794f32a68454566355d7d2
Yong Zhang
64794f32a68454566355d7d3
Yingqing He
64794f32a68454566355d7d4
Hanyuan Liu
admin_assigned
2023-06-05T11:59:05.215Z
641297a1ac08ffb70793fd04
https://cdn-avatars.huggingface.co/v1/production/uploads/641297a1ac08ffb70793fd04/3XVQYehMCGwIJvqqcQ3Gf.png
Hanyuan Liu
user
hyliu
64794f32a68454566355d7d5
Haoxin Chen
claimed_verified
2024-01-19T09:24:16.621Z
6305cb2f435ec751b723d80a
/avatars/5e8519d9814038726bf98f4104f74568.svg
Haoxin Chen
user
paulai
64794f32a68454566355d7d6
Xiaodong Cun
admin_assigned
2023-06-05T12:00:24.093Z
63184c517ca1b876d99b7e0e
https://cdn-avatars.huggingface.co/v1/production/uploads/63184c517ca1b876d99b7e0e/b-qDExoeJuDXK0cJBZKnz.jpeg
Xiaodong Cun
user
vinthony
64794f32a68454566355d7d7
Xintao Wang
admin_assigned
2023-06-05T12:00:56.785Z
60e272ca6c78a8c122b12127
https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
Xintao Wang
user
Xintao
64794f32a68454566355d7d8
Ying Shan
64794f32a68454566355d7d9
Tien-Tsin Wong
Make-Your-Video: Customized Video Generation Using Textual and
  Structural Guidance
Creating a vivid video from the event or scenario in our imagination is a
truly fascinating experience. Recent advancements in text-to-video synthesis
have unveiled the potential to achieve this with prompts only. While text is
convenient in conveying the overall scene context, it may be insufficient to
control precisely. In this paper, we explore customized video generation by
utilizing text as context description and motion structure (e.g. frame-wise
depth) as concrete guidance. Our method, dubbed Make-Your-Video, involves
joint-conditional video generation using a Latent Diffusion Model that is
pre-trained for still image synthesis and then promoted for video generation
with the introduction of temporal modules. This two-stage learning scheme not
only reduces the computing resources required, but also improves the
performance by transferring the rich concepts available in image datasets
solely into video generation. Moreover, we use a simple yet effective causal
attention mask strategy to enable longer video synthesis, which mitigates the
potential quality degradation effectively. Experimental results show the
superiority of our method over existing baselines, particularly in terms of
temporal coherence and fidelity to users' guidance. In addition, our model
enables several intriguing applications that demonstrate potential for
practical usage.
Make-Your-Video generates high-quality videos by combining text descriptions with motion structure using a Latent Diffusion Model with temporal modules, achieving better performance and temporal coherence than existing methods.
Latent Diffusion Model joint-conditional video generation pre-trained temporal modules causal attention mask