Vivid-ZOO: Multi-View Video Generation with Diffusion Model
While diffusion models have shown impressive performance in 2D image/video
generation, diffusion-based Text-to-Multi-view-Video (T2MVid) generation
remains underexplored. The new challenges posed by T2MVid generation lie in the
lack of massive captioned multi-view videos and the complexity of modeling such
multi-dimensional distribution. To this end, we propose a novel diffusion-based
pipeline that generates high-quality multi-view videos centered around a
dynamic 3D object from text. Specifically, we factor the T2MVid problem into
viewpoint-space and time components. Such factorization allows us to combine
and reuse layers of advanced pre-trained multi-view image and 2D video
diffusion models to ensure multi-view consistency as well as temporal coherence
for the generated multi-view videos, largely reducing the training cost. We
further introduce alignment modules to align the latent spaces of layers from
the pre-trained multi-view and the 2D video diffusion models, addressing the
reused layers' incompatibility that arises from the domain gap between 2D and
multi-view data. In support of this and future research, we further contribute
a captioned multi-view video dataset. Experimental results demonstrate that our
method generates high-quality multi-view videos, exhibiting vivid motions,
temporal coherence, and multi-view consistency, given a variety of text
prompts.
666ddc3b2c11d1cf29d2aa91
Bing Li
claimed_verified
2024-06-15T21:10:03.603Z
666ddb45c0f3d5afc27e85ba
/avatars/dce98fc77bd8cf9e348f2d91bc3c0225.svg
Bing Li
user
bing-li-ai
666ddc3b2c11d1cf29d2aa92
Cheng Zheng
666ddc3b2c11d1cf29d2aa93
Wenxuan Zhu
666ddc3b2c11d1cf29d2aa94
Jinjie Mai
666ddc3b2c11d1cf29d2aa95
Biao Zhang
claimed_verified
2024-11-26T21:21:35.943Z
63f89d53b0ae1748524c1b8c
/avatars/850ffb2724869dbb0deba50f6c71cb95.svg
Biao Zhang
user
Zbalpha
666ddc3b2c11d1cf29d2aa96
Peter Wonka
666ddc3b2c11d1cf29d2aa97
Bernard Ghanem
Vivid-ZOO: Multi-View Video Generation with Diffusion Model
While diffusion models have shown impressive performance in 2D image/video
generation, diffusion-based Text-to-Multi-view-Video (T2MVid) generation
remains underexplored. The new challenges posed by T2MVid generation lie in the
lack of massive captioned multi-view videos and the complexity of modeling such
multi-dimensional distribution. To this end, we propose a novel diffusion-based
pipeline that generates high-quality multi-view videos centered around a
dynamic 3D object from text. Specifically, we factor the T2MVid problem into
viewpoint-space and time components. Such factorization allows us to combine
and reuse layers of advanced pre-trained multi-view image and 2D video
diffusion models to ensure multi-view consistency as well as temporal coherence
for the generated multi-view videos, largely reducing the training cost. We
further introduce alignment modules to align the latent spaces of layers from
the pre-trained multi-view and the 2D video diffusion models, addressing the
reused layers' incompatibility that arises from the domain gap between 2D and
multi-view data. In support of this and future research, we further contribute
a captioned multi-view video dataset. Experimental results demonstrate that our
method generates high-quality multi-view videos, exhibiting vivid motions,
temporal coherence, and multi-view consistency, given a variety of text
prompts.
A novel diffusion-based pipeline generates high-quality multi-view videos from text by leveraging pre-trained models and addressing compatibility issues between 2D and multi-view data.
diffusion-based Text-to-Multi-view-Video T2MVid generation viewpoint-space time components pre-trained multi-view image 2D video diffusion models multi-view consistency temporal coherence alignment modules latent spaces domain gap captioned multi-view video dataset