TalkinNeRF: Animatable Neural Fields for Full-Body Talking Humans
We introduce a novel framework that learns a dynamic neural radiance field
(NeRF) for full-body talking humans from monocular videos. Prior work
represents only the body pose or the face. However, humans communicate with
their full body, combining body pose, hand gestures, as well as facial
expressions. In this work, we propose TalkinNeRF, a unified NeRF-based network
that represents the holistic 4D human motion. Given a monocular video of a
subject, we learn corresponding modules for the body, face, and hands, that are
combined together to generate the final result. To capture complex finger
articulation, we learn an additional deformation field for the hands. Our
multi-identity representation enables simultaneous training for multiple
subjects, as well as robust animation under completely unseen poses. It can
also generalize to novel identities, given only a short video as input. We
demonstrate state-of-the-art performance for animating full-body talking
humans, with fine-grained hand articulation and facial expressions.
66f4c08a279ccb2be378d6c6
Aggelina Chatziagapi
admin_assigned
2024-09-26T10:31:10.377Z
66ef430a133a4761fa261e78
/avatars/fecf5806ff97d86688e310004a04cf96.svg
Aggelina Chatziagapi
user
aggelina
66f4c08a279ccb2be378d6c7
Bindita Chaudhuri
66f4c08a279ccb2be378d6c8
Amit Kumar
66f4c08a279ccb2be378d6c9
Rakesh Ranjan
66f4c08a279ccb2be378d6ca
Dimitris Samaras
66f4c08a279ccb2be378d6cb
Nikolaos Sarafianos
admin_assigned
2024-09-26T10:33:08.169Z
66999bab3197305664859a8e
/avatars/199f92ccdf0397c13bfbe69595bcb8e1.svg
Nikolaos Sarafianos
user
nsarafianos
TalkinNeRF: Animatable Neural Fields for Full-Body Talking Humans
We introduce a novel framework that learns a dynamic neural radiance field
(NeRF) for full-body talking humans from monocular videos. Prior work
represents only the body pose or the face. However, humans communicate with
their full body, combining body pose, hand gestures, as well as facial
expressions. In this work, we propose TalkinNeRF, a unified NeRF-based network
that represents the holistic 4D human motion. Given a monocular video of a
subject, we learn corresponding modules for the body, face, and hands, that are
combined together to generate the final result. To capture complex finger
articulation, we learn an additional deformation field for the hands. Our
multi-identity representation enables simultaneous training for multiple
subjects, as well as robust animation under completely unseen poses. It can
also generalize to novel identities, given only a short video as input. We
demonstrate state-of-the-art performance for animating full-body talking
humans, with fine-grained hand articulation and facial expressions.
TalkinNeRF is a novel NeRF-based framework that captures full-body human motion and communicates information through body pose, hand gestures, and facial expressions using a unified model.
dynamic neural radiance field NeRF full-body talking humans monocular videos body pose hand gestures facial expressions unified NeRF-based network holistic 4D human motion deformation field multi-identity representation