ZeroAvatar: Zero-shot 3D Avatar Generation from a Single Image
Recent advancements in text-to-image generation have enabled significant
progress in zero-shot 3D shape generation. This is achieved by score
distillation, a methodology that uses pre-trained text-to-image diffusion
models to optimize the parameters of a 3D neural presentation, e.g. Neural
Radiance Field (NeRF). While showing promising results, existing methods are
often not able to preserve the geometry of complex shapes, such as human
bodies. To address this challenge, we present ZeroAvatar, a method that
introduces the explicit 3D human body prior to the optimization process.
Specifically, we first estimate and refine the parameters of a parametric human
body from a single image. Then during optimization, we use the posed parametric
body as additional geometry constraint to regularize the diffusion model as
well as the underlying density field. Lastly, we propose a UV-guided texture
regularization term to further guide the completion of texture on invisible
body parts. We show that ZeroAvatar significantly enhances the robustness and
3D consistency of optimization-based image-to-3D avatar generation,
outperforming existing zero-shot image-to-3D methods.
647412babf9b32c6bbcce340
Zhenzhen Weng
claimed_verified
2023-05-31T07:24:20.475Z
630834fbf0dc38fb47c245b2
/avatars/579d083fdbfba7331b07941f0bef83a4.svg
Zhenzhen Weng
user
zzweng
647412babf9b32c6bbcce341
Zeyu Wang
647412babf9b32c6bbcce342
Serena Yeung
ZeroAvatar: Zero-shot 3D Avatar Generation from a Single Image
Recent advancements in text-to-image generation have enabled significant
progress in zero-shot 3D shape generation. This is achieved by score
distillation, a methodology that uses pre-trained text-to-image diffusion
models to optimize the parameters of a 3D neural presentation, e.g. Neural
Radiance Field (NeRF). While showing promising results, existing methods are
often not able to preserve the geometry of complex shapes, such as human
bodies. To address this challenge, we present ZeroAvatar, a method that
introduces the explicit 3D human body prior to the optimization process.
Specifically, we first estimate and refine the parameters of a parametric human
body from a single image. Then during optimization, we use the posed parametric
body as additional geometry constraint to regularize the diffusion model as
well as the underlying density field. Lastly, we propose a UV-guided texture
regularization term to further guide the completion of texture on invisible
body parts. We show that ZeroAvatar significantly enhances the robustness and
3D consistency of optimization-based image-to-3D avatar generation,
outperforming existing zero-shot image-to-3D methods.
ZeroAvatar enhances zero-shot image-to-3D avatar generation by integrating a 3D human body prior, parameter estimation, and UV-guided texture regularization to improve geometry and texture consistency.
score distillation text-to-image diffusion models Neural Radiance Field NeRF parametric human body diffusion model density field UV-guided texture regularization optimization-based image-to-3D avatar generation