IllumiNeRF: 3D Relighting without Inverse Rendering
Existing methods for relightable view synthesis -- using a set of images of
an object under unknown lighting to recover a 3D representation that can be
rendered from novel viewpoints under a target illumination -- are based on
inverse rendering, and attempt to disentangle the object geometry, materials,
and lighting that explain the input images. Furthermore, this typically
involves optimization through differentiable Monte Carlo rendering, which is
brittle and computationally-expensive. In this work, we propose a simpler
approach: we first relight each input image using an image diffusion model
conditioned on lighting and then reconstruct a Neural Radiance Field (NeRF)
with these relit images, from which we render novel views under the target
lighting. We demonstrate that this strategy is surprisingly competitive and
achieves state-of-the-art results on multiple relighting benchmarks. Please see
our project page at https://illuminerf.github.io/.
6667cac4576792d8883870cd
Xiaoming Zhao
admin_assigned
2024-06-11T07:46:00.557Z
642006ed290342c5df8158cc
/avatars/408d35ff2512eb93ed6570befa33aa03.svg
Xiaoming Zhao
user
xzhao
6667cac4576792d8883870ce
Pratul P. Srinivasan
6667cac4576792d8883870cf
Dor Verbin
admin_assigned
2024-06-11T07:46:18.538Z
6315b0fc915d0b806830b424
/avatars/e25a7c8df2364c6e8024ec5c646a2889.svg
Dor Verbin
user
dorverbin
6667cac4576792d8883870d0
Keunhong Park
6667cac4576792d8883870d1
Ricardo Martin Brualla
6667cac4576792d8883870d2
Philipp Henzler
IllumiNeRF: 3D Relighting without Inverse Rendering
Existing methods for relightable view synthesis -- using a set of images of
an object under unknown lighting to recover a 3D representation that can be
rendered from novel viewpoints under a target illumination -- are based on
inverse rendering, and attempt to disentangle the object geometry, materials,
and lighting that explain the input images. Furthermore, this typically
involves optimization through differentiable Monte Carlo rendering, which is
brittle and computationally-expensive. In this work, we propose a simpler
approach: we first relight each input image using an image diffusion model
conditioned on lighting and then reconstruct a Neural Radiance Field (NeRF)
with these relit images, from which we render novel views under the target
lighting. We demonstrate that this strategy is surprisingly competitive and
achieves state-of-the-art results on multiple relighting benchmarks. Please see
our project page at https://illuminerf.github.io/.
A method for relightable view synthesis uses image diffusion models to relight input images before reconstructing a Neural Radiance Field, achieving state-of-the-art results.
diffusion models Neural Radiance Field NeRF