VIMI: Grounding Video Generation through Multi-modal Instruction
Existing text-to-video diffusion models rely solely on text-only encoders for
their pretraining. This limitation stems from the absence of large-scale
multimodal prompt video datasets, resulting in a lack of visual grounding and
restricting their versatility and application in multimodal integration. To
address this, we construct a large-scale multimodal prompt dataset by employing
retrieval methods to pair in-context examples with the given text prompts and
then utilize a two-stage training strategy to enable diverse video generation
tasks within the same model. In the first stage, we propose a multimodal
conditional video generation framework for pretraining on these augmented
datasets, establishing a foundational model for grounded video generation.
Secondly, we finetune the model from the first stage on three video generation
tasks, incorporating multi-modal instructions. This process further refines the
model's ability to handle diverse inputs and tasks, ensuring seamless
integration of multi-modal information. After this two-stage train-ing process,
VIMI demonstrates multimodal understanding capabilities, producing contextually
rich and personalized videos grounded in the provided inputs, as shown in
Figure 1. Compared to previous visual grounded video generation methods, VIMI
can synthesize consistent and temporally coherent videos with large motion
while retaining the semantic control. Lastly, VIMI also achieves
state-of-the-art text-to-video generation results on UCF101 benchmark.
668df26e3d0bc7c475c42380
Yuwei Fang
admin_assigned
2024-07-10T10:48:35.613Z
6406d1dcc3ab325efa9db198
/avatars/781f3cd455596f7cffa0906cc487f79b.svg
Yuwei Fang
user
studyfang
668df26e3d0bc7c475c42381
Willi Menapace
admin_assigned
2024-07-10T10:48:41.447Z
6315358a362e3e95ea538081
/avatars/3b089a25a87c2e83c6b23ccb5d2dc73e.svg
Willi Menapace
user
willi-menapace
668df26e3d0bc7c475c42382
Aliaksandr Siarohin
admin_assigned
2024-07-10T10:48:46.539Z
64276311eb9a0ed86180715b
/avatars/76f933cd549f10e5e2db379de235d304.svg
Aliaksandr Siarohin
user
aliaksandr-siarohin
668df26e3d0bc7c475c42383
Tsai-Shien Chen
admin_assigned
2024-07-10T10:48:51.931Z
6330f570b68c7453d2ee8945
/avatars/5b394a72f277aa6c886b21b477010ab0.svg
Tsai-Shien Chen
user
tschen
668df26e3d0bc7c475c42384
Kuan-Chien Wang
668df26e3d0bc7c475c42385
Ivan Skorokhodov
admin_assigned
2024-07-10T10:49:11.388Z
63610db13c7147fae7de88e3
/avatars/d7e97a16cfee39e1e50d7a5b747876f1.svg
Ivan Skorokhodov
user
universome
668df26e3d0bc7c475c42386
Graham Neubig
admin_assigned
2024-07-10T10:49:17.329Z
60de14638bedd2315529d43f
https://cdn-avatars.huggingface.co/v1/production/uploads/1625166923504-noauth.png
Graham Neubig
user
gneubig
668df26e3d0bc7c475c42387
Sergey Tulyakov
VIMI: Grounding Video Generation through Multi-modal Instruction
Existing text-to-video diffusion models rely solely on text-only encoders for
their pretraining. This limitation stems from the absence of large-scale
multimodal prompt video datasets, resulting in a lack of visual grounding and
restricting their versatility and application in multimodal integration. To
address this, we construct a large-scale multimodal prompt dataset by employing
retrieval methods to pair in-context examples with the given text prompts and
then utilize a two-stage training strategy to enable diverse video generation
tasks within the same model. In the first stage, we propose a multimodal
conditional video generation framework for pretraining on these augmented
datasets, establishing a foundational model for grounded video generation.
Secondly, we finetune the model from the first stage on three video generation
tasks, incorporating multi-modal instructions. This process further refines the
model's ability to handle diverse inputs and tasks, ensuring seamless
integration of multi-modal information. After this two-stage train-ing process,
VIMI demonstrates multimodal understanding capabilities, producing contextually
rich and personalized videos grounded in the provided inputs, as shown in
Figure 1. Compared to previous visual grounded video generation methods, VIMI
can synthesize consistent and temporally coherent videos with large motion
while retaining the semantic control. Lastly, VIMI also achieves
state-of-the-art text-to-video generation results on UCF101 benchmark.
VIMI, a two-stage trained multimodal conditional video generation framework, enhances text-to-video diffusion models with visual grounding through cross-modal datasets and instructions, achieving state-of-the-art results.
text-to-video diffusion models multimodal prompt dataset retrieval methods two-stage training multimodal conditional video generation finetuning multi-modal instructions grounded video generation contextually rich temporally coherent UCF101 benchmark