Text2Control3D: Controllable 3D Avatar Generation in Neural Radiance
  Fields using Geometry-Guided Text-to-Image Diffusion Model
Recent advances in diffusion models such as ControlNet have enabled
geometrically controllable, high-fidelity text-to-image generation. However,
none of them addresses the question of adding such controllability to
text-to-3D generation. In response, we propose Text2Control3D, a controllable
text-to-3D avatar generation method whose facial expression is controllable
given a monocular video casually captured with hand-held camera. Our main
strategy is to construct the 3D avatar in Neural Radiance Fields (NeRF)
optimized with a set of controlled viewpoint-aware images that we generate from
ControlNet, whose condition input is the depth map extracted from the input
video. When generating the viewpoint-aware images, we utilize cross-reference
attention to inject well-controlled, referential facial expression and
appearance via cross attention. We also conduct low-pass filtering of Gaussian
latent of the diffusion model in order to ameliorate the viewpoint-agnostic
texture problem we observed from our empirical analysis, where the
viewpoint-aware images contain identical textures on identical pixel positions
that are incomprehensible in 3D. Finally, to train NeRF with the images that
are viewpoint-aware yet are not strictly consistent in geometry, our approach
considers per-image geometric variation as a view of deformation from a shared
3D canonical space. Consequently, we construct the 3D avatar in a canonical
space of deformable NeRF by learning a set of per-image deformation via
deformation field table. We demonstrate the empirical results and discuss the
effectiveness of our method.
64fa8cfde0dc35986b83ce50
Sungwon Hwang
claimed_verified
2023-09-08T09:33:14.311Z
642fcfc0a043f0ac7deeaae0
/avatars/6cc46dd480cdc0d86c7a509e22782a13.svg
Sungwon Hwang
user
sungwon95
64fa8cfde0dc35986b83ce51
Junha Hyung
admin_assigned
2023-09-08T12:25:19.461Z
6331ed964db0a767bbee4706
/avatars/2ef3fa6083ca5208f4b87c91345267d9.svg
Junha Hyung
user
JunhaH
64fa8cfde0dc35986b83ce52
Jaegul Choo
admin_assigned
2023-09-08T12:25:28.411Z
64be3127805e5b64572da65c
/avatars/edd3e94ba9e375827cc75b164602bcac.svg
Jaegul Choo
user
joyfull78
Text2Control3D: Controllable 3D Avatar Generation in Neural Radiance
  Fields using Geometry-Guided Text-to-Image Diffusion Model
Recent advances in diffusion models such as ControlNet have enabled
geometrically controllable, high-fidelity text-to-image generation. However,
none of them addresses the question of adding such controllability to
text-to-3D generation. In response, we propose Text2Control3D, a controllable
text-to-3D avatar generation method whose facial expression is controllable
given a monocular video casually captured with hand-held camera. Our main
strategy is to construct the 3D avatar in Neural Radiance Fields (NeRF)
optimized with a set of controlled viewpoint-aware images that we generate from
ControlNet, whose condition input is the depth map extracted from the input
video. When generating the viewpoint-aware images, we utilize cross-reference
attention to inject well-controlled, referential facial expression and
appearance via cross attention. We also conduct low-pass filtering of Gaussian
latent of the diffusion model in order to ameliorate the viewpoint-agnostic
texture problem we observed from our empirical analysis, where the
viewpoint-aware images contain identical textures on identical pixel positions
that are incomprehensible in 3D. Finally, to train NeRF with the images that
are viewpoint-aware yet are not strictly consistent in geometry, our approach
considers per-image geometric variation as a view of deformation from a shared
3D canonical space. Consequently, we construct the 3D avatar in a canonical
space of deformable NeRF by learning a set of per-image deformation via
deformation field table. We demonstrate the empirical results and discuss the
effectiveness of our method.
Neural fields Diffusion models ControlNet Geometrically controllable High-fidelity text-to-image generation Text-to-3D generation Controllable text-to-3D avatar generation Facial expression controllability Monocular video Neural Radiance Fields (NeRF) Controlled viewpoint-aware images Depth map Cross-reference attention Gaussian latent Viewpoint-agnostic texture problem Per-image geometric variation View deformation from a shared 3D canonical space Deformable NeRF Per-image deformation Deformation field table