StableDreamer: Taming Noisy Score Distillation Sampling for Text-to-3D
In the realm of text-to-3D generation, utilizing 2D diffusion models through
score distillation sampling (SDS) frequently leads to issues such as blurred
appearances and multi-faced geometry, primarily due to the intrinsically noisy
nature of the SDS loss. Our analysis identifies the core of these challenges as
the interaction among noise levels in the 2D diffusion process, the
architecture of the diffusion network, and the 3D model representation. To
overcome these limitations, we present StableDreamer, a methodology
incorporating three advances. First, inspired by InstructNeRF2NeRF, we
formalize the equivalence of the SDS generative prior and a simple supervised
L2 reconstruction loss. This finding provides a novel tool to debug SDS, which
we use to show the impact of time-annealing noise levels on reducing
multi-faced geometries. Second, our analysis shows that while image-space
diffusion contributes to geometric precision, latent-space diffusion is crucial
for vivid color rendition. Based on this observation, StableDreamer introduces
a two-stage training strategy that effectively combines these aspects,
resulting in high-fidelity 3D models. Third, we adopt an anisotropic 3D
Gaussians representation, replacing Neural Radiance Fields (NeRFs), to enhance
the overall quality, reduce memory usage during training, and accelerate
rendering speeds, and better capture semi-transparent objects. StableDreamer
reduces multi-face geometries, generates fine details, and converges stably.
656fdc8234440bdf2ed850d9
Pengsheng Guo
656fdc8234440bdf2ed850da
Hans Hao
656fdc8234440bdf2ed850db
Adam Caccavale
656fdc8234440bdf2ed850dc
Zhongzheng Ren
656fdc8234440bdf2ed850dd
Edward Zhang
656fdc8234440bdf2ed850de
Qi Shan
656fdc8234440bdf2ed850df
Aditya Sankar
656fdc8234440bdf2ed850e0
Alexander G. Schwing
656fdc8234440bdf2ed850e1
Alex Colburn
656fdc8234440bdf2ed850e2
Fangchang Ma
claimed_verified
2023-12-06T10:25:02.754Z
656ea9c84b5b695353709f0d
/avatars/0be91edfbfe9ab9a83325e2d844801f0.svg
Fangchang Ma
user
fangchangma
StableDreamer: Taming Noisy Score Distillation Sampling for Text-to-3D
In the realm of text-to-3D generation, utilizing 2D diffusion models through
score distillation sampling (SDS) frequently leads to issues such as blurred
appearances and multi-faced geometry, primarily due to the intrinsically noisy
nature of the SDS loss. Our analysis identifies the core of these challenges as
the interaction among noise levels in the 2D diffusion process, the
architecture of the diffusion network, and the 3D model representation. To
overcome these limitations, we present StableDreamer, a methodology
incorporating three advances. First, inspired by InstructNeRF2NeRF, we
formalize the equivalence of the SDS generative prior and a simple supervised
L2 reconstruction loss. This finding provides a novel tool to debug SDS, which
we use to show the impact of time-annealing noise levels on reducing
multi-faced geometries. Second, our analysis shows that while image-space
diffusion contributes to geometric precision, latent-space diffusion is crucial
for vivid color rendition. Based on this observation, StableDreamer introduces
a two-stage training strategy that effectively combines these aspects,
resulting in high-fidelity 3D models. Third, we adopt an anisotropic 3D
Gaussians representation, replacing Neural Radiance Fields (NeRFs), to enhance
the overall quality, reduce memory usage during training, and accelerate
rendering speeds, and better capture semi-transparent objects. StableDreamer
reduces multi-face geometries, generates fine details, and converges stably.
StableDreamer addresses the issues in 2D diffusion model-based text-to-3D generation by combining image-space and latent-space diffusion, using anisotropic 3D Gaussians, and improving noise management, leading to high-fidelity 3D models.
score distillation sampling SDS 2D diffusion models 3D model representation diffusion network InstructNeRF2NeRF supervised L2 reconstruction loss time-annealing noise levels image-space diffusion latent-space diffusion anisotropic 3D Gaussians Neural Radiance Fields NeRFs