Text2Control3D: Controllable 3D Avatar Generation in Neural Radiance
  Fields using Geometry-Guided Text-to-Image Diffusion Model
Recent advances in diffusion models such as ControlNet have enabled
geometrically controllable, high-fidelity text-to-image generation. However,
none of them addresses the question of adding such controllability to
text-to-3D generation. In response, we propose Text2Control3D, a controllable
text-to-3D avatar generation method whose facial expression is controllable
given a monocular video casually captured with hand-held camera. Our main
strategy is to construct the 3D avatar in Neural Radiance Fields (NeRF)
optimized with a set of controlled viewpoint-aware images that we generate from
ControlNet, whose condition input is the depth map extracted from the input
video. When generating the viewpoint-aware images, we utilize cross-reference
attention to inject well-controlled, referential facial expression and
appearance via cross attention. We also conduct low-pass filtering of Gaussian
latent of the diffusion model in order to ameliorate the viewpoint-agnostic
texture problem we observed from our empirical analysis, where the
viewpoint-aware images contain identical textures on identical pixel positions
that are incomprehensible in 3D. Finally, to train NeRF with the images that
are viewpoint-aware yet are not strictly consistent in geometry, our approach
considers per-image geometric variation as a view of deformation from a shared
3D canonical space. Consequently, we construct the 3D avatar in a canonical
space of deformable NeRF by learning a set of per-image deformation via
deformation field table. We demonstrate the empirical results and discuss the
effectiveness of our method.
64fa8cfde0dc35986b83ce50
Sungwon Hwang
claimed_verified
2023-09-08T09:33:14.311Z
642fcfc0a043f0ac7deeaae0
/avatars/6cc46dd480cdc0d86c7a509e22782a13.svg
Sungwon Hwang
user
sungwon95
64fa8cfde0dc35986b83ce51
Junha Hyung
admin_assigned
2023-09-08T12:25:19.461Z
6331ed964db0a767bbee4706
/avatars/2ef3fa6083ca5208f4b87c91345267d9.svg
Junha Hyung
user
JunhaH
64fa8cfde0dc35986b83ce52
Jaegul Choo
admin_assigned
2023-09-08T12:25:28.411Z
64be3127805e5b64572da65c
/avatars/edd3e94ba9e375827cc75b164602bcac.svg
Jaegul Choo
user
joyfull78
Text2Control3D: Controllable 3D Avatar Generation in Neural Radiance
  Fields using Geometry-Guided Text-to-Image Diffusion Model
Recent advances in diffusion models such as ControlNet have enabled
geometrically controllable, high-fidelity text-to-image generation. However,
none of them addresses the question of adding such controllability to
text-to-3D generation. In response, we propose Text2Control3D, a controllable
text-to-3D avatar generation method whose facial expression is controllable
given a monocular video casually captured with hand-held camera. Our main
strategy is to construct the 3D avatar in Neural Radiance Fields (NeRF)
optimized with a set of controlled viewpoint-aware images that we generate from
ControlNet, whose condition input is the depth map extracted from the input
video. When generating the viewpoint-aware images, we utilize cross-reference
attention to inject well-controlled, referential facial expression and
appearance via cross attention. We also conduct low-pass filtering of Gaussian
latent of the diffusion model in order to ameliorate the viewpoint-agnostic
texture problem we observed from our empirical analysis, where the
viewpoint-aware images contain identical textures on identical pixel positions
that are incomprehensible in 3D. Finally, to train NeRF with the images that
are viewpoint-aware yet are not strictly consistent in geometry, our approach
considers per-image geometric variation as a view of deformation from a shared
3D canonical space. Consequently, we construct the 3D avatar in a canonical
space of deformable NeRF by learning a set of per-image deformation via
deformation field table. We demonstrate the empirical results and discuss the
effectiveness of our method.
Text2Control3D generates controllable 3D avatars from monocular videos using diffusion models, Neural Radiance Fields, and cross-reference attention.
ControlNet Neural Radiance Fields NeRF viewpoint-aware images cross-reference attention Gaussian latent viewpoint-agnostic texture deformation field table deformable NeRF