A Diffusion Approach to Radiance Field Relighting using
  Multi-Illumination Synthesis
Relighting radiance fields is severely underconstrained for multi-view data,
which is most often captured under a single illumination condition; It is
especially hard for full scenes containing multiple objects. We introduce a
method to create relightable radiance fields using such single-illumination
data by exploiting priors extracted from 2D image diffusion models. We first
fine-tune a 2D diffusion model on a multi-illumination dataset conditioned by
light direction, allowing us to augment a single-illumination capture into a
realistic -- but possibly inconsistent -- multi-illumination dataset from
directly defined light directions. We use this augmented data to create a
relightable radiance field represented by 3D Gaussian splats. To allow direct
control of light direction for low-frequency lighting, we represent appearance
with a multi-layer perceptron parameterized on light direction. To enforce
multi-view consistency and overcome inaccuracies we optimize a per-image
auxiliary feature vector. We show results on synthetic and real multi-view data
under single illumination, demonstrating that our method successfully exploits
2D diffusion model priors to allow realistic 3D relighting for complete scenes.
Project site
https://repo-sam.inria.fr/fungraph/generative-radiance-field-relighting/
66e8381119f4837cdfe210c4
Yohan Poirier-Ginter
admin_assigned
2024-09-17T07:19:01.667Z
637b48da56db0404b7c64b20
/avatars/059b869881aefb230df72d3282eeab1b.svg
Yohan Poirier-Ginter
user
yohanpg
66e8381119f4837cdfe210c5
Alban Gauthier
admin_assigned
2024-09-17T07:19:07.774Z
634474ca6e988c6a792cfead
/avatars/87fea91613c0854cc0f134e067eb47cd.svg
Alban Gauthier
user
albangwut
66e8381119f4837cdfe210c6
Julien Phillip
claimed_verified
2024-09-17T15:08:18.255Z
63e0dbb35ba41def8794dab8
/avatars/38a28103b15d93c7d9243bc5418e623e.svg
Julien Philip
user
Jovphi
66e8381119f4837cdfe210c7
Jean-Francois Lalonde
claimed_verified
2024-09-17T15:08:16.450Z
66e9905f00d5cd69e36e0e1a
/avatars/082eb98a634ae2f46c3664c2e6d331a1.svg
Jean-Francois Lalonde
user
jflalonde
66e8381119f4837cdfe210c8
George Drettakis
claimed_verified
2024-09-18T11:34:43.792Z
66eaa4ba1654494d77f28d42
/avatars/42e6d78bc6c32ea43dad0684b210c815.svg
George Drettakis
user
gdrett
A Diffusion Approach to Radiance Field Relighting using
  Multi-Illumination Synthesis
Relighting radiance fields is severely underconstrained for multi-view data,
which is most often captured under a single illumination condition; It is
especially hard for full scenes containing multiple objects. We introduce a
method to create relightable radiance fields using such single-illumination
data by exploiting priors extracted from 2D image diffusion models. We first
fine-tune a 2D diffusion model on a multi-illumination dataset conditioned by
light direction, allowing us to augment a single-illumination capture into a
realistic -- but possibly inconsistent -- multi-illumination dataset from
directly defined light directions. We use this augmented data to create a
relightable radiance field represented by 3D Gaussian splats. To allow direct
control of light direction for low-frequency lighting, we represent appearance
with a multi-layer perceptron parameterized on light direction. To enforce
multi-view consistency and overcome inaccuracies we optimize a per-image
auxiliary feature vector. We show results on synthetic and real multi-view data
under single illumination, demonstrating that our method successfully exploits
2D diffusion model priors to allow realistic 3D relighting for complete scenes.
Project site
https://repo-sam.inria.fr/fungraph/generative-radiance-field-relighting/
The method creates relightable 3D radiance fields using single-illumination data by leveraging 2D diffusion model priors, achieving realistic relighting in full scenes.
radiance fields 2D diffusion models multi-illumination Gaussian splats multi-layer perceptron per-image auxiliary feature vector multi-view consistency