Sherpa3D: Boosting High-Fidelity Text-to-3D Generation via Coarse 3D
  Prior
Recently, 3D content creation from text prompts has demonstrated remarkable
progress by utilizing 2D and 3D diffusion models. While 3D diffusion models
ensure great multi-view consistency, their ability to generate high-quality and
diverse 3D assets is hindered by the limited 3D data. In contrast, 2D diffusion
models find a distillation approach that achieves excellent generalization and
rich details without any 3D data. However, 2D lifting methods suffer from
inherent view-agnostic ambiguity thereby leading to serious multi-face Janus
issues, where text prompts fail to provide sufficient guidance to learn
coherent 3D results. Instead of retraining a costly viewpoint-aware model, we
study how to fully exploit easily accessible coarse 3D knowledge to enhance the
prompts and guide 2D lifting optimization for refinement. In this paper, we
propose Sherpa3D, a new text-to-3D framework that achieves high-fidelity,
generalizability, and geometric consistency simultaneously. Specifically, we
design a pair of guiding strategies derived from the coarse 3D prior generated
by the 3D diffusion model: a structural guidance for geometric fidelity and a
semantic guidance for 3D coherence. Employing the two types of guidance, the 2D
diffusion model enriches the 3D content with diversified and high-quality
results. Extensive experiments show the superiority of our Sherpa3D over the
state-of-the-art text-to-3D methods in terms of quality and 3D consistency.
6577f42b1e4ac8627f96fe50
Fangfu Liu
admin_assigned
2023-12-12T08:14:52.412Z
6505a02f9310ce8c400edc63
/avatars/bbf781594fc8c812316711aa8e2797aa.svg
Fangfu Liu
user
Liuff23
6577f42b1e4ac8627f96fe51
Diankun Wu
admin_assigned
2023-12-12T08:15:01.479Z
6577fba2eb02736add6377f5
/avatars/3e486dd36021feaa4a0259cd89c1eee9.svg
Wu
user
Diankun
6577f42b1e4ac8627f96fe52
Yi Wei
6577f42b1e4ac8627f96fe53
Yongming Rao
admin_assigned
2023-12-12T08:15:19.905Z
63e4865354f51ea342d45d78
/avatars/2e7eccc878751331ca8b282f53e38899.svg
Yongming Rao
user
raoyongming
6577f42b1e4ac8627f96fe54
Yueqi Duan
Sherpa3D: Boosting High-Fidelity Text-to-3D Generation via Coarse 3D
  Prior
Recently, 3D content creation from text prompts has demonstrated remarkable
progress by utilizing 2D and 3D diffusion models. While 3D diffusion models
ensure great multi-view consistency, their ability to generate high-quality and
diverse 3D assets is hindered by the limited 3D data. In contrast, 2D diffusion
models find a distillation approach that achieves excellent generalization and
rich details without any 3D data. However, 2D lifting methods suffer from
inherent view-agnostic ambiguity thereby leading to serious multi-face Janus
issues, where text prompts fail to provide sufficient guidance to learn
coherent 3D results. Instead of retraining a costly viewpoint-aware model, we
study how to fully exploit easily accessible coarse 3D knowledge to enhance the
prompts and guide 2D lifting optimization for refinement. In this paper, we
propose Sherpa3D, a new text-to-3D framework that achieves high-fidelity,
generalizability, and geometric consistency simultaneously. Specifically, we
design a pair of guiding strategies derived from the coarse 3D prior generated
by the 3D diffusion model: a structural guidance for geometric fidelity and a
semantic guidance for 3D coherence. Employing the two types of guidance, the 2D
diffusion model enriches the 3D content with diversified and high-quality
results. Extensive experiments show the superiority of our Sherpa3D over the
state-of-the-art text-to-3D methods in terms of quality and 3D consistency.
Sherpa3D leverages 2D diffusion models and coarse 3D priors to generate high-fidelity, geometrically consistent, and diverse 3D content from text prompts.
3D diffusion models 2D diffusion models multi-view consistency view-agnostic ambiguity multi-face Janus issues coarse 3D knowledge text-to-3D framework structural guidance semantic guidance geometric fidelity 3D coherence