Learning Disentangled Avatars with Hybrid 3D Representations
Tremendous efforts have been made to learn animatable and photorealistic
human avatars. Towards this end, both explicit and implicit 3D representations
are heavily studied for a holistic modeling and capture of the whole human
(e.g., body, clothing, face and hair), but neither representation is an optimal
choice in terms of representation efficacy since different parts of the human
avatar have different modeling desiderata. For example, meshes are generally
not suitable for modeling clothing and hair. Motivated by this, we present
Disentangled Avatars~(DELTA), which models humans with hybrid explicit-implicit
3D representations. DELTA takes a monocular RGB video as input, and produces a
human avatar with separate body and clothing/hair layers. Specifically, we
demonstrate two important applications for DELTA. For the first one, we
consider the disentanglement of the human body and clothing and in the second,
we disentangle the face and hair. To do so, DELTA represents the body or face
with an explicit mesh-based parametric 3D model and the clothing or hair with
an implicit neural radiance field. To make this possible, we design an
end-to-end differentiable renderer that integrates meshes into volumetric
rendering, enabling DELTA to learn directly from monocular videos without any
3D supervision. Finally, we show that how these two applications can be easily
combined to model full-body avatars, such that the hair, face, body and
clothing can be fully disentangled yet jointly rendered. Such a disentanglement
enables hair and clothing transfer to arbitrary body shapes. We empirically
validate the effectiveness of DELTA's disentanglement by demonstrating its
promising performance on disentangled reconstruction, virtual clothing try-on
and hairstyle transfer. To facilitate future research, we also release an
open-sourced pipeline for the study of hybrid human avatar modeling.
6501391635ec971762ad7d76
Yao Feng
6501391635ec971762ad7d77
Weiyang Liu
admin_assigned
2023-09-14T08:23:03.768Z
648905d1a15c43c791d4381f
https://cdn-avatars.huggingface.co/v1/production/uploads/648905d1a15c43c791d4381f/GpqGBzsLiMHX0gWZEz3qn.jpeg
Weiyang Liu
user
wy1iu
6501391635ec971762ad7d78
Timo Bolkart
admin_assigned
2023-09-14T08:23:11.707Z
648b73eb1c3e38f18eb39988
/avatars/ac7b858f504648c1f5d1c3e3034c7996.svg
Timo Bolkart
user
tbolkart
6501391635ec971762ad7d79
Jinlong Yang
6501391635ec971762ad7d7a
Marc Pollefeys
claimed_verified
2025-02-25T09:42:59.411Z
67b5fa179782a5e2fd2cb26a
/avatars/62c38f29ec641e001eeddf840bea21a0.svg
Marc Pollefeys
user
mapo1
6501391635ec971762ad7d7b
Michael J. Black
admin_assigned
2023-09-14T08:24:01.868Z
649fbababa7a05e3626546c8
/avatars/a9aa4b4914649527094fc1b12779e014.svg
Michael J. Black
user
MichaelJBlack
Learning Disentangled Avatars with Hybrid 3D Representations
Tremendous efforts have been made to learn animatable and photorealistic
human avatars. Towards this end, both explicit and implicit 3D representations
are heavily studied for a holistic modeling and capture of the whole human
(e.g., body, clothing, face and hair), but neither representation is an optimal
choice in terms of representation efficacy since different parts of the human
avatar have different modeling desiderata. For example, meshes are generally
not suitable for modeling clothing and hair. Motivated by this, we present
Disentangled Avatars~(DELTA), which models humans with hybrid explicit-implicit
3D representations. DELTA takes a monocular RGB video as input, and produces a
human avatar with separate body and clothing/hair layers. Specifically, we
demonstrate two important applications for DELTA. For the first one, we
consider the disentanglement of the human body and clothing and in the second,
we disentangle the face and hair. To do so, DELTA represents the body or face
with an explicit mesh-based parametric 3D model and the clothing or hair with
an implicit neural radiance field. To make this possible, we design an
end-to-end differentiable renderer that integrates meshes into volumetric
rendering, enabling DELTA to learn directly from monocular videos without any
3D supervision. Finally, we show that how these two applications can be easily
combined to model full-body avatars, such that the hair, face, body and
clothing can be fully disentangled yet jointly rendered. Such a disentanglement
enables hair and clothing transfer to arbitrary body shapes. We empirically
validate the effectiveness of DELTA's disentanglement by demonstrating its
promising performance on disentangled reconstruction, virtual clothing try-on
and hairstyle transfer. To facilitate future research, we also release an
open-sourced pipeline for the study of hybrid human avatar modeling.
Disentangled Avatars (DELTA) uses hybrid explicit-implicit 3D representations to model disentangled human body, clothing, face, and hair from monocular videos, enabling applications like virtual try-on and hairstyle transfer.
explicit-implicit 3D representations monocular RGB video end-to-end differentiable renderer volumetric rendering disentangled reconstruction virtual clothing try-on hairstyle transfer