Droplet3D: Commonsense Priors from Videos Facilitate 3D Generation
Scaling laws have validated the success and promise of large-data-trained
models in creative generation across text, image, and video domains. However,
this paradigm faces data scarcity in the 3D domain, as there is far less of it
available on the internet compared to the aforementioned modalities.
Fortunately, there exist adequate videos that inherently contain commonsense
priors, offering an alternative supervisory signal to mitigate the
generalization bottleneck caused by limited native 3D data. On the one hand,
videos capturing multiple views of an object or scene provide a spatial
consistency prior for 3D generation. On the other hand, the rich semantic
information contained within the videos enables the generated content to be
more faithful to the text prompts and semantically plausible. This paper
explores how to apply the video modality in 3D asset generation, spanning
datasets to models. We introduce Droplet3D-4M, the first large-scale video
dataset with multi-view level annotations, and train Droplet3D, a generative
model supporting both image and dense text input. Extensive experiments
validate the effectiveness of our approach, demonstrating its ability to
produce spatially consistent and semantically plausible content. Moreover, in
contrast to the prevailing 3D solutions, our approach exhibits the potential
for extension to scene-level applications. This indicates that the commonsense
priors from the videos significantly facilitate 3D creation. We have
open-sourced all resources including the dataset, code, technical framework,
and model weights: https://dropletx.github.io/.
68b50e02851c6e7b001eca38
Xiaochuan Li
claimed_verified
2025-09-01T07:50:33.701Z
66b01dc4e48856bb718f2ba8
https://cdn-avatars.huggingface.co/v1/production/uploads/66b01dc4e48856bb718f2ba8/MHZyIDd3BEtsnH9M2BvzM.jpeg
Xiaochuan Li
user
lixiaochuan
68b50e02851c6e7b001eca39
Guoguang Du
68b50e02851c6e7b001eca3a
Runze Zhang
68b50e02851c6e7b001eca3b
Liang Jin
68b50e02851c6e7b001eca3c
Qi Jia
68b50e02851c6e7b001eca3d
Lihua Lu
68b50e02851c6e7b001eca3e
Zhenhua Guo
68b50e02851c6e7b001eca3f
Yaqian Zhao
68b50e02851c6e7b001eca40
Haiyang Liu
68b50e02851c6e7b001eca41
Tianqi Wang
68b50e02851c6e7b001eca42
Changsheng Li
68b50e02851c6e7b001eca43
Xiaoli Gong
68b50e02851c6e7b001eca44
Rengang Li
68b50e02851c6e7b001eca45
Baoyu Fan
Droplet3D: Commonsense Priors from Videos Facilitate 3D Generation
Scaling laws have validated the success and promise of large-data-trained
models in creative generation across text, image, and video domains. However,
this paradigm faces data scarcity in the 3D domain, as there is far less of it
available on the internet compared to the aforementioned modalities.
Fortunately, there exist adequate videos that inherently contain commonsense
priors, offering an alternative supervisory signal to mitigate the
generalization bottleneck caused by limited native 3D data. On the one hand,
videos capturing multiple views of an object or scene provide a spatial
consistency prior for 3D generation. On the other hand, the rich semantic
information contained within the videos enables the generated content to be
more faithful to the text prompts and semantically plausible. This paper
explores how to apply the video modality in 3D asset generation, spanning
datasets to models. We introduce Droplet3D-4M, the first large-scale video
dataset with multi-view level annotations, and train Droplet3D, a generative
model supporting both image and dense text input. Extensive experiments
validate the effectiveness of our approach, demonstrating its ability to
produce spatially consistent and semantically plausible content. Moreover, in
contrast to the prevailing 3D solutions, our approach exhibits the potential
for extension to scene-level applications. This indicates that the commonsense
priors from the videos significantly facilitate 3D creation. We have
open-sourced all resources including the dataset, code, technical framework,
and model weights: https://dropletx.github.io/.
Using video data to provide commonsense priors enhances 3D asset generation, enabling spatial consistency and semantic plausibility in 3D content creation.
scaling laws large-data-trained models creative generation 3D domain data scarcity commonsense priors supervisory signal generalization bottleneck spatial consistency prior semantic information Droplet3D-4M Droplet3D generative model image input dense text input scene-level applications