Controllable Dynamic Appearance for Neural 3D Portraits
Recent advances in Neural Radiance Fields (NeRFs) have made it possible to
reconstruct and reanimate dynamic portrait scenes with control over head-pose,
facial expressions and viewing direction. However, training such models assumes
photometric consistency over the deformed region e.g. the face must be evenly
lit as it deforms with changing head-pose and facial expression. Such
photometric consistency across frames of a video is hard to maintain, even in
studio environments, thus making the created reanimatable neural portraits
prone to artifacts during reanimation. In this work, we propose CoDyNeRF, a
system that enables the creation of fully controllable 3D portraits in
real-world capture conditions. CoDyNeRF learns to approximate illumination
dependent effects via a dynamic appearance model in the canonical space that is
conditioned on predicted surface normals and the facial expressions and
head-pose deformations. The surface normals prediction is guided using 3DMM
normals that act as a coarse prior for the normals of the human head, where
direct prediction of normals is hard due to rigid and non-rigid deformations
induced by head-pose and facial expression changes. Using only a
smartphone-captured short video of a subject for training, we demonstrate the
effectiveness of our method on free view synthesis of a portrait scene with
explicit head pose and expression controls, and realistic lighting effects. The
project page can be found here:
http://shahrukhathar.github.io/2023/08/22/CoDyNeRF.html
650bd01d1f2949b99e65d252
ShahRukh Athar
claimed_verified
2023-09-21T13:17:37.470Z
650c3a2bd439bbbbadf40436
/avatars/a5a7c04a03580ae59c28eb6f90d43b47.svg
ShahRukh Athar
user
Sathar
650bd01d1f2949b99e65d253
Zhixin Shu
admin_assigned
2023-09-21T10:10:44.639Z
62a8efd508a7ea93ff18785a
/avatars/ff259233d437833a304329bb973a5a04.svg
Zhixin Shu
user
zhixinshu
650bd01d1f2949b99e65d254
Zexiang Xu
claimed_verified
2024-10-21T07:55:38.136Z
65565b1b8616d8d740abb976
/avatars/c245b19ed05292bf1bf4a9d8cf2a409c.svg
Zexiang Xu
user
zexiangxu
650bd01d1f2949b99e65d255
Fuji Luan
650bd01d1f2949b99e65d256
Sai Bi
650bd01d1f2949b99e65d257
Kalyan Sunkavalli
650bd01d1f2949b99e65d258
Dimitris Samaras
Controllable Dynamic Appearance for Neural 3D Portraits
Recent advances in Neural Radiance Fields (NeRFs) have made it possible to
reconstruct and reanimate dynamic portrait scenes with control over head-pose,
facial expressions and viewing direction. However, training such models assumes
photometric consistency over the deformed region e.g. the face must be evenly
lit as it deforms with changing head-pose and facial expression. Such
photometric consistency across frames of a video is hard to maintain, even in
studio environments, thus making the created reanimatable neural portraits
prone to artifacts during reanimation. In this work, we propose CoDyNeRF, a
system that enables the creation of fully controllable 3D portraits in
real-world capture conditions. CoDyNeRF learns to approximate illumination
dependent effects via a dynamic appearance model in the canonical space that is
conditioned on predicted surface normals and the facial expressions and
head-pose deformations. The surface normals prediction is guided using 3DMM
normals that act as a coarse prior for the normals of the human head, where
direct prediction of normals is hard due to rigid and non-rigid deformations
induced by head-pose and facial expression changes. Using only a
smartphone-captured short video of a subject for training, we demonstrate the
effectiveness of our method on free view synthesis of a portrait scene with
explicit head pose and expression controls, and realistic lighting effects. The
project page can be found here:
http://shahrukhathar.github.io/2023/08/22/CoDyNeRF.html
CoDyNeRF enables the creation of photometrically consistent, fully controllable 3D portraits from smartphone videos, accounting for dynamic lighting and facial movements.
Neural Radiance Fields CoDyNeRF photometric consistency dynamic appearance model canonical space surface normals 3DMM normals head-pose facial expressions free view synthesis realistic lighting effects