UniSDF: Unifying Neural Representations for High-Fidelity 3D
  Reconstruction of Complex Scenes with Reflections
Neural 3D scene representations have shown great potential for 3D
reconstruction from 2D images. However, reconstructing real-world captures of
complex scenes still remains a challenge. Existing generic 3D reconstruction
methods often struggle to represent fine geometric details and do not
adequately model reflective surfaces of large-scale scenes. Techniques that
explicitly focus on reflective surfaces can model complex and detailed
reflections by exploiting better reflection parameterizations. However, we
observe that these methods are often not robust in real unbounded scenarios
where non-reflective as well as reflective components are present. In this
work, we propose UniSDF, a general purpose 3D reconstruction method that can
reconstruct large complex scenes with reflections. We investigate both
view-based as well as reflection-based color prediction parameterization
techniques and find that explicitly blending these representations in 3D space
enables reconstruction of surfaces that are more geometrically accurate,
especially for reflective surfaces. We further combine this representation with
a multi-resolution grid backbone that is trained in a coarse-to-fine manner,
enabling faster reconstructions than prior methods. Extensive experiments on
object-level datasets DTU, Shiny Blender as well as unbounded datasets Mip-NeRF
360 and Ref-NeRF real demonstrate that our method is able to robustly
reconstruct complex large-scale scenes with fine details and reflective
surfaces. Please see our project page at
https://fangjinhuawang.github.io/UniSDF.
6583bce1af09ca206bcda61b
Fangjinhua Wang
claimed_verified
2023-12-21T09:56:37.576Z
6584099fcce29a06215a5839
/avatars/8bc526b953f8ddf91a79cf210ee1c4bd.svg
Fangjinhua Wang
user
fangjinhuawang
6583bce1af09ca206bcda61c
Marie-Julie Rakotosaona
6583bce1af09ca206bcda61d
Michael Niemeyer
admin_assigned
2023-12-21T09:40:41.228Z
63f4f16171a5d395c71d2d1a
/avatars/0377c4e6b35f8bd1a29d38c99a5943c1.svg
Michael Niemeyer
user
mniemeyer
6583bce1af09ca206bcda61e
Richard Szeliski
6583bce1af09ca206bcda61f
Marc Pollefeys
claimed_verified
2025-02-25T09:43:30.609Z
67b5fa179782a5e2fd2cb26a
/avatars/62c38f29ec641e001eeddf840bea21a0.svg
Marc Pollefeys
user
mapo1
6583bce1af09ca206bcda620
Federico Tombari
UniSDF: Unifying Neural Representations for High-Fidelity 3D
  Reconstruction of Complex Scenes with Reflections
Neural 3D scene representations have shown great potential for 3D
reconstruction from 2D images. However, reconstructing real-world captures of
complex scenes still remains a challenge. Existing generic 3D reconstruction
methods often struggle to represent fine geometric details and do not
adequately model reflective surfaces of large-scale scenes. Techniques that
explicitly focus on reflective surfaces can model complex and detailed
reflections by exploiting better reflection parameterizations. However, we
observe that these methods are often not robust in real unbounded scenarios
where non-reflective as well as reflective components are present. In this
work, we propose UniSDF, a general purpose 3D reconstruction method that can
reconstruct large complex scenes with reflections. We investigate both
view-based as well as reflection-based color prediction parameterization
techniques and find that explicitly blending these representations in 3D space
enables reconstruction of surfaces that are more geometrically accurate,
especially for reflective surfaces. We further combine this representation with
a multi-resolution grid backbone that is trained in a coarse-to-fine manner,
enabling faster reconstructions than prior methods. Extensive experiments on
object-level datasets DTU, Shiny Blender as well as unbounded datasets Mip-NeRF
360 and Ref-NeRF real demonstrate that our method is able to robustly
reconstruct complex large-scale scenes with fine details and reflective
surfaces. Please see our project page at
https://fangjinhuawang.github.io/UniSDF.
UniSDF reconstructs complex 3D scenes with reflections by blending view-based and reflection-based color prediction techniques with a multi-resolution, coarse-to-fine grid backbone.
Neural 3D 3D reconstruction 2D images fine geometric details reflective surfaces reflection parameterizations reflection-based color prediction view-based color prediction 3D space multi-resolution grid backbone coarse-to-fine DTU Shiny Blender Mip-NeRF 360 Ref-NeRF real