DiffusionGAN3D: Boosting Text-guided 3D Generation and Domain Adaption
  by Combining 3D GANs and Diffusion Priors
Text-guided domain adaption and generation of 3D-aware portraits find many
applications in various fields. However, due to the lack of training data and
the challenges in handling the high variety of geometry and appearance, the
existing methods for these tasks suffer from issues like inflexibility,
instability, and low fidelity. In this paper, we propose a novel framework
DiffusionGAN3D, which boosts text-guided 3D domain adaption and generation by
combining 3D GANs and diffusion priors. Specifically, we integrate the
pre-trained 3D generative models (e.g., EG3D) and text-to-image diffusion
models. The former provides a strong foundation for stable and high-quality
avatar generation from text. And the diffusion models in turn offer powerful
priors and guide the 3D generator finetuning with informative direction to
achieve flexible and efficient text-guided domain adaption. To enhance the
diversity in domain adaption and the generation capability in text-to-avatar,
we introduce the relative distance loss and case-specific learnable triplane
respectively. Besides, we design a progressive texture refinement module to
improve the texture quality for both tasks above. Extensive experiments
demonstrate that the proposed framework achieves excellent results in both
domain adaption and text-to-avatar tasks, outperforming existing methods in
terms of generation quality and efficiency. The project homepage is at
https://younglbw.github.io/DiffusionGAN3D-homepage/.
658e4279bea632dc942f5655
Biwen Lei
658e4279bea632dc942f5656
Kai Yu
658e4279bea632dc942f5657
Mengyang Feng
658e4279bea632dc942f5658
Miaomiao Cui
658e4279bea632dc942f5659
Xuansong Xie
DiffusionGAN3D: Boosting Text-guided 3D Generation and Domain Adaption
  by Combining 3D GANs and Diffusion Priors
Text-guided domain adaption and generation of 3D-aware portraits find many
applications in various fields. However, due to the lack of training data and
the challenges in handling the high variety of geometry and appearance, the
existing methods for these tasks suffer from issues like inflexibility,
instability, and low fidelity. In this paper, we propose a novel framework
DiffusionGAN3D, which boosts text-guided 3D domain adaption and generation by
combining 3D GANs and diffusion priors. Specifically, we integrate the
pre-trained 3D generative models (e.g., EG3D) and text-to-image diffusion
models. The former provides a strong foundation for stable and high-quality
avatar generation from text. And the diffusion models in turn offer powerful
priors and guide the 3D generator finetuning with informative direction to
achieve flexible and efficient text-guided domain adaption. To enhance the
diversity in domain adaption and the generation capability in text-to-avatar,
we introduce the relative distance loss and case-specific learnable triplane
respectively. Besides, we design a progressive texture refinement module to
improve the texture quality for both tasks above. Extensive experiments
demonstrate that the proposed framework achieves excellent results in both
domain adaption and text-to-avatar tasks, outperforming existing methods in
terms of generation quality and efficiency. The project homepage is at
https://younglbw.github.io/DiffusionGAN3D-homepage/.
A new framework, DiffusionGAN3D, enhances 3D domain adaptation and avatar generation by combining 3D GANs with diffusion priors, improving quality and efficiency through relative distance loss and a progressive texture refinement module.
3D GANs diffusion priors EG3D text-to-image diffusion models relative distance loss triplane progressive texture refinement module domain adaption text-to-avatar generation quality efficiency