Decompositional Neural Scene Reconstruction with Generative Diffusion
  Prior
Decompositional reconstruction of 3D scenes, with complete shapes and
detailed texture of all objects within, is intriguing for downstream
applications but remains challenging, particularly with sparse views as input.
Recent approaches incorporate semantic or geometric regularization to address
this issue, but they suffer significant degradation in underconstrained areas
and fail to recover occluded regions. We argue that the key to solving this
problem lies in supplementing missing information for these areas. To this end,
we propose DP-Recon, which employs diffusion priors in the form of Score
Distillation Sampling (SDS) to optimize the neural representation of each
individual object under novel views. This provides additional information for
the underconstrained areas, but directly incorporating diffusion prior raises
potential conflicts between the reconstruction and generative guidance.
Therefore, we further introduce a visibility-guided approach to dynamically
adjust the per-pixel SDS loss weights. Together these components enhance both
geometry and appearance recovery while remaining faithful to input images.
Extensive experiments across Replica and ScanNet++ demonstrate that our method
significantly outperforms SOTA methods. Notably, it achieves better object
reconstruction under 10 views than the baselines under 100 views. Our method
enables seamless text-based editing for geometry and appearance through SDS
optimization and produces decomposed object meshes with detailed UV maps that
support photorealistic Visual effects (VFX) editing. The project page is
available at https://dp-recon.github.io/.
67dc367858db060fe5a82916
Junfeng Ni
claimed_verified
2025-03-21T11:41:52.218Z
65ae5edddacd99fd58277620
https://cdn-avatars.huggingface.co/v1/production/uploads/65ae5edddacd99fd58277620/Yq2adEKtkxaSzdEhCfpnY.jpeg
Junfeng Ni
user
JunfengNi
67dc367858db060fe5a82917
Yu Liu
67dc367858db060fe5a82918
Ruijie Lu
67dc367858db060fe5a82919
Zirui Zhou
67dc367858db060fe5a8291a
Song-Chun Zhu
67dc367858db060fe5a8291b
Yixin Chen
67dc367858db060fe5a8291c
Siyuan Huang
Decompositional Neural Scene Reconstruction with Generative Diffusion
  Prior
Decompositional reconstruction of 3D scenes, with complete shapes and
detailed texture of all objects within, is intriguing for downstream
applications but remains challenging, particularly with sparse views as input.
Recent approaches incorporate semantic or geometric regularization to address
this issue, but they suffer significant degradation in underconstrained areas
and fail to recover occluded regions. We argue that the key to solving this
problem lies in supplementing missing information for these areas. To this end,
we propose DP-Recon, which employs diffusion priors in the form of Score
Distillation Sampling (SDS) to optimize the neural representation of each
individual object under novel views. This provides additional information for
the underconstrained areas, but directly incorporating diffusion prior raises
potential conflicts between the reconstruction and generative guidance.
Therefore, we further introduce a visibility-guided approach to dynamically
adjust the per-pixel SDS loss weights. Together these components enhance both
geometry and appearance recovery while remaining faithful to input images.
Extensive experiments across Replica and ScanNet++ demonstrate that our method
significantly outperforms SOTA methods. Notably, it achieves better object
reconstruction under 10 views than the baselines under 100 views. Our method
enables seamless text-based editing for geometry and appearance through SDS
optimization and produces decomposed object meshes with detailed UV maps that
support photorealistic Visual effects (VFX) editing. The project page is
available at https://dp-recon.github.io/.
DP-Recon uses diffusion priors with Score Distillation Sampling to enhance 3D scene reconstruction from sparse views, recovering underconstrained areas more accurately than existing methods and enabling text-based editing.
diffusion priors Score Distillation Sampling DP-Recon visibility-guided approach SDS loss weights geometric reconstruction appearance recovery Replica ScanNet++ photorealistic Visual effects (VFX)