MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequence
Recent advancements in video generation have primarily leveraged diffusion
models for short-duration content. However, these approaches often fall short
in modeling complex narratives and maintaining character consistency over
extended periods, which is essential for long-form video production like
movies. We propose MovieDreamer, a novel hierarchical framework that integrates
the strengths of autoregressive models with diffusion-based rendering to
pioneer long-duration video generation with intricate plot progressions and
high visual fidelity. Our approach utilizes autoregressive models for global
narrative coherence, predicting sequences of visual tokens that are
subsequently transformed into high-quality video frames through diffusion
rendering. This method is akin to traditional movie production processes, where
complex stories are factorized down into manageable scene capturing. Further,
we employ a multimodal script that enriches scene descriptions with detailed
character information and visual style, enhancing continuity and character
identity across scenes. We present extensive experiments across various movie
genres, demonstrating that our approach not only achieves superior visual and
narrative quality but also effectively extends the duration of generated
content significantly beyond current capabilities. Homepage:
https://aim-uofa.github.io/MovieDreamer/.
66a05dafcf52b98e1d2873e8
Canyu Zhao
admin_assigned
2024-07-24T08:06:36.821Z
646efd223dd912a539e0bd46
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/EOFAv5xvOgJOzuDgh4nSb.png
Canyu Zhao
user
Canyu
66a05dafcf52b98e1d2873e9
Mingyu Liu
admin_assigned
2024-07-24T08:06:54.688Z
652e25d2e647b0ee0a024f26
/avatars/b5c65cf6c8d0ddc9b8ef0226e0295d56.svg
Mingyu Liu
user
MingyuLiu
66a05dafcf52b98e1d2873ea
Wen Wang
claimed_verified
2024-10-28T11:28:49.379Z
63f089456309c84d5f47f951
/avatars/04b926a7f2ad091ee00fef0c59903492.svg
Wen Wang
user
wwen1997
66a05dafcf52b98e1d2873eb
Jianlong Yuan
admin_assigned
2024-07-24T08:07:29.462Z
66684c82473b56ef2d7c340b
/avatars/9158f3f2c0e287c26d6910ea14edaa3a.svg
Yuan Jian-Long
user
OxyOksirotl
66a05dafcf52b98e1d2873ec
Hao Chen
66a05dafcf52b98e1d2873ed
Bo Zhang
claimed_verified
2024-10-21T07:53:52.959Z
6237f3eb4cf571bb1ed4036b
/avatars/cbfb8ddda18e65d18a818db9f176e16e.svg
Bo Zhang
user
zhangmozhe
66a05dafcf52b98e1d2873ee
Chunhua Shen
MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequence
Recent advancements in video generation have primarily leveraged diffusion
models for short-duration content. However, these approaches often fall short
in modeling complex narratives and maintaining character consistency over
extended periods, which is essential for long-form video production like
movies. We propose MovieDreamer, a novel hierarchical framework that integrates
the strengths of autoregressive models with diffusion-based rendering to
pioneer long-duration video generation with intricate plot progressions and
high visual fidelity. Our approach utilizes autoregressive models for global
narrative coherence, predicting sequences of visual tokens that are
subsequently transformed into high-quality video frames through diffusion
rendering. This method is akin to traditional movie production processes, where
complex stories are factorized down into manageable scene capturing. Further,
we employ a multimodal script that enriches scene descriptions with detailed
character information and visual style, enhancing continuity and character
identity across scenes. We present extensive experiments across various movie
genres, demonstrating that our approach not only achieves superior visual and
narrative quality but also effectively extends the duration of generated
content significantly beyond current capabilities. Homepage:
https://aim-uofa.github.io/MovieDreamer/.
MovieDreamer integrates autoregressive models with diffusion rendering to produce high-fidelity, long-form video content with coherent narratives and consistent characters.
autoregressive models diffusion rendering narrative coherence visual tokens multimodal script scene descriptions character information visual style