Bridging the Gap: Studio-like Avatar Creation from a Monocular Phone
  Capture
Creating photorealistic avatars for individuals traditionally involves
extensive capture sessions with complex and expensive studio devices like the
LightStage system. While recent strides in neural representations have enabled
the generation of photorealistic and animatable 3D avatars from quick phone
scans, they have the capture-time lighting baked-in, lack facial details and
have missing regions in areas such as the back of the ears. Thus, they lag in
quality compared to studio-captured avatars. In this paper, we propose a method
that bridges this gap by generating studio-like illuminated texture maps from
short, monocular phone captures. We do this by parameterizing the phone texture
maps using the W^+ space of a StyleGAN2, enabling near-perfect
reconstruction. Then, we finetune a StyleGAN2 by sampling in the W^+
parameterized space using a very small set of studio-captured textures as an
adversarial training signal. To further enhance the realism and accuracy of
facial details, we super-resolve the output of the StyleGAN2 using carefully
designed diffusion model that is guided by image gradients of the
phone-captured texture map. Once trained, our method excels at producing
studio-like facial texture maps from casual monocular smartphone videos.
Demonstrating its capabilities, we showcase the generation of photorealistic,
uniformly lit, complete avatars from monocular phone captures.
http://shahrukhathar.github.io/2024/07/22/Bridging.html{The project page
can be found here.}
66a85d31b812afa43f9a83b3
ShahRukh Athar
admin_assigned
2024-07-30T09:05:24.124Z
650c3a2bd439bbbbadf40436
/avatars/a5a7c04a03580ae59c28eb6f90d43b47.svg
ShahRukh Athar
user
Sathar
66a85d31b812afa43f9a83b4
Shunsuke Saito
extracted_confirmed
2024-08-23T13:33:23.377Z
630688edd37ce67e0e50e548
/avatars/1681e465d7649f67f94e1b69d236cb1e.svg
Shunsuke Saito
user
psyth
66a85d31b812afa43f9a83b5
Zhengyu Yang
66a85d31b812afa43f9a83b6
Stanislav Pidhorsky
66a85d31b812afa43f9a83b7
Chen Cao
admin_assigned
2024-07-30T09:06:15.742Z
66a606790ab6a8dc67910b6c
/avatars/d4b55d788b69efdfe0be5a1afe571d51.svg
cao
user
chencao95
Bridging the Gap: Studio-like Avatar Creation from a Monocular Phone
  Capture
Creating photorealistic avatars for individuals traditionally involves
extensive capture sessions with complex and expensive studio devices like the
LightStage system. While recent strides in neural representations have enabled
the generation of photorealistic and animatable 3D avatars from quick phone
scans, they have the capture-time lighting baked-in, lack facial details and
have missing regions in areas such as the back of the ears. Thus, they lag in
quality compared to studio-captured avatars. In this paper, we propose a method
that bridges this gap by generating studio-like illuminated texture maps from
short, monocular phone captures. We do this by parameterizing the phone texture
maps using the W^+ space of a StyleGAN2, enabling near-perfect
reconstruction. Then, we finetune a StyleGAN2 by sampling in the W^+
parameterized space using a very small set of studio-captured textures as an
adversarial training signal. To further enhance the realism and accuracy of
facial details, we super-resolve the output of the StyleGAN2 using carefully
designed diffusion model that is guided by image gradients of the
phone-captured texture map. Once trained, our method excels at producing
studio-like facial texture maps from casual monocular smartphone videos.
Demonstrating its capabilities, we showcase the generation of photorealistic,
uniformly lit, complete avatars from monocular phone captures.
http://shahrukhathar.github.io/2024/07/22/Bridging.html{The project page
can be found here.}
A method is proposed to generate studio-quality facial texture maps from monocular phone captures using StyleGAN2 and diffusion models, enabling photorealistic avatar creation.
StyleGAN2 $W^+$ space diffusion model image gradients