Compositional 3D-aware Video Generation with LLM Director
Significant progress has been made in text-to-video generation through the
use of powerful generative models and large-scale internet data. However,
substantial challenges remain in precisely controlling individual concepts
within the generated video, such as the motion and appearance of specific
characters and the movement of viewpoints. In this work, we propose a novel
paradigm that generates each concept in 3D representation separately and then
composes them with priors from Large Language Models (LLM) and 2D diffusion
models. Specifically, given an input textual prompt, our scheme consists of
three stages: 1) We leverage LLM as the director to first decompose the complex
query into several sub-prompts that indicate individual concepts within the
video~(e.g., scene, objects, motions), then we let LLM to invoke
pre-trained expert models to obtain corresponding 3D representations of
concepts. 2) To compose these representations, we prompt multi-modal LLM to
produce coarse guidance on the scales and coordinates of trajectories for the
objects. 3) To make the generated frames adhere to natural image distribution,
we further leverage 2D diffusion priors and use Score Distillation Sampling to
refine the composition. Extensive experiments demonstrate that our method can
generate high-fidelity videos from text with diverse motion and flexible
control over each concept. Project page: https://aka.ms/c3v.
66d7ef327e6c9509bb09f617
Hanxin Zhu
claimed_verified
2024-09-04T07:15:00.377Z
650e5beff874d950df2203f0
/avatars/384b35ea59b55f8d454be9efb7cce47c.svg
Hanxin ZHU
user
ingzzzz
66d7ef327e6c9509bb09f618
Tianyu He
extracted_confirmed
2024-09-04T05:28:42.906Z
619b7b1cab4c7b7f16a7d59e
https://cdn-avatars.huggingface.co/v1/production/uploads/619b7b1cab4c7b7f16a7d59e/6TvXaAqBghAMYO1-j5l4v.jpeg
Tianyu He
user
deeptimhe
66d7ef327e6c9509bb09f619
Anni Tang
66d7ef327e6c9509bb09f61a
Junliang Guo
extracted_pending
2024-09-04T05:25:07.682Z
66868659ccb9539da85c4e14
/avatars/515a49363872c23d57a6f75063606348.svg
Junliang Guo
user
leo-guo
66d7ef327e6c9509bb09f61b
Zhibo Chen
66d7ef327e6c9509bb09f61c
Jiang Bian
admin_assigned
2024-09-04T08:38:09.562Z
63f253f8f4e30ffd2bd308fb
/avatars/303f4c7ee588f638acf78a7966786e1e.svg
Jiang Bian
user
bianjiang
Compositional 3D-aware Video Generation with LLM Director
Significant progress has been made in text-to-video generation through the
use of powerful generative models and large-scale internet data. However,
substantial challenges remain in precisely controlling individual concepts
within the generated video, such as the motion and appearance of specific
characters and the movement of viewpoints. In this work, we propose a novel
paradigm that generates each concept in 3D representation separately and then
composes them with priors from Large Language Models (LLM) and 2D diffusion
models. Specifically, given an input textual prompt, our scheme consists of
three stages: 1) We leverage LLM as the director to first decompose the complex
query into several sub-prompts that indicate individual concepts within the
video~(e.g., scene, objects, motions), then we let LLM to invoke
pre-trained expert models to obtain corresponding 3D representations of
concepts. 2) To compose these representations, we prompt multi-modal LLM to
produce coarse guidance on the scales and coordinates of trajectories for the
objects. 3) To make the generated frames adhere to natural image distribution,
we further leverage 2D diffusion priors and use Score Distillation Sampling to
refine the composition. Extensive experiments demonstrate that our method can
generate high-fidelity videos from text with diverse motion and flexible
control over each concept. Project page: https://aka.ms/c3v.
A novel text-to-video generation method decomposes concepts into 3D representations using LLMs, composes them with multi-modal LLM guidance, and refines with 2D diffusion models to produce high-fidelity videos with controlled motion and flexibility.
generative models large-scale internet data text-to-video generation Large Language Models LLM 3D representation 2D diffusion models Score Distillation Sampling