ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with
  Variational Score Distillation
Score distillation sampling (SDS) has shown great promise in text-to-3D
generation by distilling pretrained large-scale text-to-image diffusion models,
but suffers from over-saturation, over-smoothing, and low-diversity problems.
In this work, we propose to model the 3D parameter as a random variable instead
of a constant as in SDS and present variational score distillation (VSD), a
principled particle-based variational framework to explain and address the
aforementioned issues in text-to-3D generation. We show that SDS is a special
case of VSD and leads to poor samples with both small and large CFG weights. In
comparison, VSD works well with various CFG weights as ancestral sampling from
diffusion models and simultaneously improves the diversity and sample quality
with a common CFG weight (i.e., 7.5). We further present various improvements
in the design space for text-to-3D such as distillation time schedule and
density initialization, which are orthogonal to the distillation algorithm yet
not well explored. Our overall approach, dubbed ProlificDreamer, can generate
high rendering resolution (i.e., 512times512) and high-fidelity NeRF with
rich structure and complex effects (e.g., smoke and drops). Further,
initialized from NeRF, meshes fine-tuned by VSD are meticulously detailed and
photo-realistic. Project page: https://ml.cs.tsinghua.edu.cn/prolificdreamer/
64701c0a850a938d6c568531
Zhengyi Wang
claimed_verified
2023-05-26T13:58:00.955Z
634e15aec1ce28f1de91c470
https://cdn-avatars.huggingface.co/v1/production/uploads/634e15aec1ce28f1de91c470/MdtJKgE_fbHydhqESr-DH.png
Zhengyi Wang
user
Zhengyi
64701c0a850a938d6c568532
Cheng Lu
extracted_confirmed
2023-05-26T03:39:57.857Z
6308ab64e3246e2be8884f4f
/avatars/7d8002caeacf27258996f31e4a2e186f.svg
LuChengTHU
user
LuChengTHU
64701c0a850a938d6c568533
Yikai Wang
extracted_confirmed
2023-05-26T02:42:51.456Z
64206b35e40f66bcd1e3c985
/avatars/a8f830d1c47702f642045399e59c9fe8.svg
Yikai Wang
user
yikaiw
64701c0a850a938d6c568534
Fan Bao
claimed_verified
2023-05-26T08:07:15.886Z
633131798ef21f47308ce49b
/avatars/1a3b5b032cbbfc1499ec96aba5260902.svg
fan bao
user
baofff
64701c0a850a938d6c568535
Chongxuan Li
64701c0a850a938d6c568536
Hang Su
64701c0a850a938d6c568537
Jun Zhu
ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with
  Variational Score Distillation
Score distillation sampling (SDS) has shown great promise in text-to-3D
generation by distilling pretrained large-scale text-to-image diffusion models,
but suffers from over-saturation, over-smoothing, and low-diversity problems.
In this work, we propose to model the 3D parameter as a random variable instead
of a constant as in SDS and present variational score distillation (VSD), a
principled particle-based variational framework to explain and address the
aforementioned issues in text-to-3D generation. We show that SDS is a special
case of VSD and leads to poor samples with both small and large CFG weights. In
comparison, VSD works well with various CFG weights as ancestral sampling from
diffusion models and simultaneously improves the diversity and sample quality
with a common CFG weight (i.e., 7.5). We further present various improvements
in the design space for text-to-3D such as distillation time schedule and
density initialization, which are orthogonal to the distillation algorithm yet
not well explored. Our overall approach, dubbed ProlificDreamer, can generate
high rendering resolution (i.e., 512times512) and high-fidelity NeRF with
rich structure and complex effects (e.g., smoke and drops). Further,
initialized from NeRF, meshes fine-tuned by VSD are meticulously detailed and
photo-realistic. Project page: https://ml.cs.tsinghua.edu.cn/prolificdreamer/
Variational score distillation addresses issues in text-to-3D generation by modeling 3D parameters as random variables, improving sample quality and diversity.
score distillation sampling variational score distillation particle-based variational framework ancestral sampling diffusion models text-to-3D generation NeRF rendering resolution high-fidelity fine-tuning ProlificDreamer