Relightify: Relightable 3D Faces from a Single Image via Diffusion
  Models
Following the remarkable success of diffusion models on image generation,
recent works have also demonstrated their impressive ability to address a
number of inverse problems in an unsupervised way, by properly constraining the
sampling process based on a conditioning input. Motivated by this, in this
paper, we present the first approach to use diffusion models as a prior for
highly accurate 3D facial BRDF reconstruction from a single image. We start by
leveraging a high-quality UV dataset of facial reflectance (diffuse and
specular albedo and normals), which we render under varying illumination
settings to simulate natural RGB textures and, then, train an unconditional
diffusion model on concatenated pairs of rendered textures and reflectance
components. At test time, we fit a 3D morphable model to the given image and
unwrap the face in a partial UV texture. By sampling from the diffusion model,
while retaining the observed texture part intact, the model inpaints not only
the self-occluded areas but also the unknown reflectance components, in a
single sequence of denoising steps. In contrast to existing methods, we
directly acquire the observed texture from the input image, thus, resulting in
more faithful and consistent reflectance estimation. Through a series of
qualitative and quantitative comparisons, we demonstrate superior performance
in both texture completion as well as reflectance reconstruction tasks.
645c3c89fbdc26a155fc80d4
Foivos Paraperas Papantoniou
claimed_verified
2023-05-11T14:59:56.390Z
645d00f703fc86c46b3e57eb
/avatars/1240d1915f051c18f31e342a423c9c24.svg
Foivos Paraperas
user
FoivosPar
645c3c89fbdc26a155fc80d5
Alexandros Lattas
claimed_verified
2023-05-12T08:10:58.459Z
645df1be8a22774edb290765
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/OduGNNzXnhkwnPkBZUw3B.jpeg
Alexandros Lattas
user
lattas
645c3c89fbdc26a155fc80d6
Stylianos Moschoglou
645c3c89fbdc26a155fc80d7
Stefanos Zafeiriou
Relightify: Relightable 3D Faces from a Single Image via Diffusion
  Models
Following the remarkable success of diffusion models on image generation,
recent works have also demonstrated their impressive ability to address a
number of inverse problems in an unsupervised way, by properly constraining the
sampling process based on a conditioning input. Motivated by this, in this
paper, we present the first approach to use diffusion models as a prior for
highly accurate 3D facial BRDF reconstruction from a single image. We start by
leveraging a high-quality UV dataset of facial reflectance (diffuse and
specular albedo and normals), which we render under varying illumination
settings to simulate natural RGB textures and, then, train an unconditional
diffusion model on concatenated pairs of rendered textures and reflectance
components. At test time, we fit a 3D morphable model to the given image and
unwrap the face in a partial UV texture. By sampling from the diffusion model,
while retaining the observed texture part intact, the model inpaints not only
the self-occluded areas but also the unknown reflectance components, in a
single sequence of denoising steps. In contrast to existing methods, we
directly acquire the observed texture from the input image, thus, resulting in
more faithful and consistent reflectance estimation. Through a series of
qualitative and quantitative comparisons, we demonstrate superior performance
in both texture completion as well as reflectance reconstruction tasks.
A novel approach uses diffusion models to reconstruct 3D facial BRDF from a single image by sampling and inpainting UV textures.
diffusion models 3D facial BRDF reconstruction UV dataset facial reflectance unconditional diffusion model rendered textures reflectance components 3D morphable model partial UV texture texture completion reflectance reconstruction