FaceCLIPNeRF: Text-driven 3D Face Manipulation using Deformable Neural
  Radiance Fields
As recent advances in Neural Radiance Fields (NeRF) have enabled
high-fidelity 3D face reconstruction and novel view synthesis, its manipulation
also became an essential task in 3D vision. However, existing manipulation
methods require extensive human labor, such as a user-provided semantic mask
and manual attribute search unsuitable for non-expert users. Instead, our
approach is designed to require a single text to manipulate a face
reconstructed with NeRF. To do so, we first train a scene manipulator, a latent
code-conditional deformable NeRF, over a dynamic scene to control a face
deformation using the latent code. However, representing a scene deformation
with a single latent code is unfavorable for compositing local deformations
observed in different instances. As so, our proposed Position-conditional
Anchor Compositor (PAC) learns to represent a manipulated scene with spatially
varying latent codes. Their renderings with the scene manipulator are then
optimized to yield high cosine similarity to a target text in CLIP embedding
space for text-driven manipulation. To the best of our knowledge, our approach
is the first to address the text-driven manipulation of a face reconstructed
with NeRF. Extensive results, comparisons, and ablation studies demonstrate the
effectiveness of our approach.
64be087f76a6e2efccb80b0c
Sungwon Hwang
admin_assigned
2023-07-24T07:42:16.208Z
642fcfc0a043f0ac7deeaae0
/avatars/6cc46dd480cdc0d86c7a509e22782a13.svg
Sungwon Hwang
user
sungwon95
64be087f76a6e2efccb80b0d
Junha Hyung
admin_assigned
2023-07-24T07:47:54.356Z
6331ed964db0a767bbee4706
/avatars/2ef3fa6083ca5208f4b87c91345267d9.svg
Junha Hyung
user
JunhaH
64be087f76a6e2efccb80b0e
Daejin Kim
64be087f76a6e2efccb80b0f
Min-Jung Kim
64be087f76a6e2efccb80b10
Jaegul Choo
claimed_verified
2023-07-24T11:31:32.437Z
64be3127805e5b64572da65c
/avatars/edd3e94ba9e375827cc75b164602bcac.svg
Jaegul Choo
user
joyfull78
FaceCLIPNeRF: Text-driven 3D Face Manipulation using Deformable Neural
  Radiance Fields
As recent advances in Neural Radiance Fields (NeRF) have enabled
high-fidelity 3D face reconstruction and novel view synthesis, its manipulation
also became an essential task in 3D vision. However, existing manipulation
methods require extensive human labor, such as a user-provided semantic mask
and manual attribute search unsuitable for non-expert users. Instead, our
approach is designed to require a single text to manipulate a face
reconstructed with NeRF. To do so, we first train a scene manipulator, a latent
code-conditional deformable NeRF, over a dynamic scene to control a face
deformation using the latent code. However, representing a scene deformation
with a single latent code is unfavorable for compositing local deformations
observed in different instances. As so, our proposed Position-conditional
Anchor Compositor (PAC) learns to represent a manipulated scene with spatially
varying latent codes. Their renderings with the scene manipulator are then
optimized to yield high cosine similarity to a target text in CLIP embedding
space for text-driven manipulation. To the best of our knowledge, our approach
is the first to address the text-driven manipulation of a face reconstructed
with NeRF. Extensive results, comparisons, and ablation studies demonstrate the
effectiveness of our approach.
Neural Radiance Fields (NeRF) 3D face reconstruction novel view synthesis semantic mask scene manipulator latent code-conditional deformable NeRF latent code Position-conditional Anchor Compositor (PAC) spatially varying latent codes CLIP embedding space text-driven manipulation