VFusion3D: Learning Scalable 3D Generative Models from Video Diffusion
  Models
This paper presents a novel paradigm for building scalable 3D generative
models utilizing pre-trained video diffusion models. The primary obstacle in
developing foundation 3D generative models is the limited availability of 3D
data. Unlike images, texts, or videos, 3D data are not readily accessible and
are difficult to acquire. This results in a significant disparity in scale
compared to the vast quantities of other types of data. To address this issue,
we propose using a video diffusion model, trained with extensive volumes of
text, images, and videos, as a knowledge source for 3D data. By unlocking its
multi-view generative capabilities through fine-tuning, we generate a
large-scale synthetic multi-view dataset to train a feed-forward 3D generative
model. The proposed model, VFusion3D, trained on nearly 3M synthetic multi-view
data, can generate a 3D asset from a single image in seconds and achieves
superior performance when compared to current SOTA feed-forward 3D generative
models, with users preferring our results over 70% of the time.
65f919e89d2f497c66f6ed00
Junlin Han
extracted_pending
2024-03-19T04:51:53.479Z
6541469f6be058da063d329a
/avatars/08683e07d89562f2fb5319854b21706c.svg
Junlin Han
user
JunlinHan
65f919e89d2f497c66f6ed01
Filippos Kokkinos
extracted_pending
2024-03-19T04:51:53.479Z
64918e6bd0aa14078dee0929
/avatars/407d6996d5d297ad3492a9a2382205cd.svg
Filippos
user
fkokkinos
65f919e89d2f497c66f6ed02
Philip Torr
VFusion3D: Learning Scalable 3D Generative Models from Video Diffusion
  Models
This paper presents a novel paradigm for building scalable 3D generative
models utilizing pre-trained video diffusion models. The primary obstacle in
developing foundation 3D generative models is the limited availability of 3D
data. Unlike images, texts, or videos, 3D data are not readily accessible and
are difficult to acquire. This results in a significant disparity in scale
compared to the vast quantities of other types of data. To address this issue,
we propose using a video diffusion model, trained with extensive volumes of
text, images, and videos, as a knowledge source for 3D data. By unlocking its
multi-view generative capabilities through fine-tuning, we generate a
large-scale synthetic multi-view dataset to train a feed-forward 3D generative
model. The proposed model, VFusion3D, trained on nearly 3M synthetic multi-view
data, can generate a 3D asset from a single image in seconds and achieves
superior performance when compared to current SOTA feed-forward 3D generative
models, with users preferring our results over 70% of the time.
A video diffusion model is fine-tuned to generate a large-scale synthetic multi-view dataset for training a feed-forward 3D generative model, achieving superior performance in 3D asset generation.
video diffusion model multi-view generative capabilities fine-tuning feed-forward 3D generative model synthetic multi-view data