SIGNeRF: Scene Integrated Generation for Neural Radiance Fields
Advances in image diffusion models have recently led to notable improvements
in the generation of high-quality images. In combination with Neural Radiance
Fields (NeRFs), they enabled new opportunities in 3D generation. However, most
generative 3D approaches are object-centric and applying them to editing
existing photorealistic scenes is not trivial. We propose SIGNeRF, a novel
approach for fast and controllable NeRF scene editing and scene-integrated
object generation. A new generative update strategy ensures 3D consistency
across the edited images, without requiring iterative optimization. We find
that depth-conditioned diffusion models inherently possess the capability to
generate 3D consistent views by requesting a grid of images instead of single
views. Based on these insights, we introduce a multi-view reference sheet of
modified images. Our method updates an image collection consistently based on
the reference sheet and refines the original NeRF with the newly generated
image set in one go. By exploiting the depth conditioning mechanism of the
image diffusion model, we gain fine control over the spatial location of the
edit and enforce shape guidance by a selected region or an external mesh.
65962815aa677adc97dc3c2d
Jan-Niklas Dihlmann
claimed_verified
2024-01-04T09:35:20.775Z
631b6dbcbf1351ed2bd05be1
https://cdn-avatars.huggingface.co/v1/production/uploads/631b6dbcbf1351ed2bd05be1/_kzcDD4qCifsb-651_LmY.png
Jan-Niklas Dihlmann
user
JDihlmann
65962815aa677adc97dc3c2e
Andreas Engelhardt
65962815aa677adc97dc3c2f
Hendrik Lensch
SIGNeRF: Scene Integrated Generation for Neural Radiance Fields
Advances in image diffusion models have recently led to notable improvements
in the generation of high-quality images. In combination with Neural Radiance
Fields (NeRFs), they enabled new opportunities in 3D generation. However, most
generative 3D approaches are object-centric and applying them to editing
existing photorealistic scenes is not trivial. We propose SIGNeRF, a novel
approach for fast and controllable NeRF scene editing and scene-integrated
object generation. A new generative update strategy ensures 3D consistency
across the edited images, without requiring iterative optimization. We find
that depth-conditioned diffusion models inherently possess the capability to
generate 3D consistent views by requesting a grid of images instead of single
views. Based on these insights, we introduce a multi-view reference sheet of
modified images. Our method updates an image collection consistently based on
the reference sheet and refines the original NeRF with the newly generated
image set in one go. By exploiting the depth conditioning mechanism of the
image diffusion model, we gain fine control over the spatial location of the
edit and enforce shape guidance by a selected region or an external mesh.
SIGNeRF, a novel method for NeRF scene editing and object generation, utilizes depth-conditioned diffusion models for 3D consistent image updates without iterative optimization.
Neural Radiance Fields NeRF depth-conditioned diffusion models 3D generation scene editing multi-view reference sheet shape guidance external mesh