Semantic Score Distillation Sampling for Compositional Text-to-3D
  Generation
Generating high-quality 3D assets from textual descriptions remains a pivotal
challenge in computer graphics and vision research. Due to the scarcity of 3D
data, state-of-the-art approaches utilize pre-trained 2D diffusion priors,
optimized through Score Distillation Sampling (SDS). Despite progress, crafting
complex 3D scenes featuring multiple objects or intricate interactions is still
difficult. To tackle this, recent methods have incorporated box or layout
guidance. However, these layout-guided compositional methods often struggle to
provide fine-grained control, as they are generally coarse and lack
expressiveness. To overcome these challenges, we introduce a novel SDS
approach, Semantic Score Distillation Sampling (SemanticSDS), designed to
effectively improve the expressiveness and accuracy of compositional text-to-3D
generation. Our approach integrates new semantic embeddings that maintain
consistency across different rendering views and clearly differentiate between
various objects and parts. These embeddings are transformed into a semantic
map, which directs a region-specific SDS process, enabling precise optimization
and compositional generation. By leveraging explicit semantic guidance, our
method unlocks the compositional capabilities of existing pre-trained diffusion
models, thereby achieving superior quality in 3D content generation,
particularly for complex objects and scenes. Experimental results demonstrate
that our SemanticSDS framework is highly effective for generating
state-of-the-art complex 3D content. Code:
https://github.com/YangLing0818/SemanticSDS-3D
670c7992f6b3c3ad2cbe2f2b
Ling Yang
admin_assigned
2024-10-14T12:35:15.951Z
64fde4e252e82dd432b74ce9
https://cdn-avatars.huggingface.co/v1/production/uploads/64fde4e252e82dd432b74ce9/-CQZbBP7FsPPyawYrsi4z.jpeg
Ling Yang
user
Lingaaaaaaa
670c7992f6b3c3ad2cbe2f2c
Zixiang Zhang
670c7992f6b3c3ad2cbe2f2d
Junlin Han
670c7992f6b3c3ad2cbe2f2e
Bohan Zeng
admin_assigned
2024-10-14T12:30:09.672Z
6671214c92412fd4640714eb
/avatars/48fa84e7bc3bb92ad0192aa26b32de10.svg
bohan zeng
user
zbhpku
670c7992f6b3c3ad2cbe2f2f
Runjia Li
admin_assigned
2024-10-14T12:29:45.711Z
638e29cf319f9c746b87ad4b
/avatars/70cac8d47847c389eb0393051a64c4a4.svg
Runjia Li
user
liguang0115
670c7992f6b3c3ad2cbe2f30
Philip Torr
admin_assigned
2024-10-14T12:23:22.107Z
6565ed28a5ec0231cb07225f
/avatars/7f95bba9aa7811d56eecb380827abfac.svg
prof philip torr
user
philiptorr
670c7992f6b3c3ad2cbe2f31
Wentao Zhang
Semantic Score Distillation Sampling for Compositional Text-to-3D
  Generation
Generating high-quality 3D assets from textual descriptions remains a pivotal
challenge in computer graphics and vision research. Due to the scarcity of 3D
data, state-of-the-art approaches utilize pre-trained 2D diffusion priors,
optimized through Score Distillation Sampling (SDS). Despite progress, crafting
complex 3D scenes featuring multiple objects or intricate interactions is still
difficult. To tackle this, recent methods have incorporated box or layout
guidance. However, these layout-guided compositional methods often struggle to
provide fine-grained control, as they are generally coarse and lack
expressiveness. To overcome these challenges, we introduce a novel SDS
approach, Semantic Score Distillation Sampling (SemanticSDS), designed to
effectively improve the expressiveness and accuracy of compositional text-to-3D
generation. Our approach integrates new semantic embeddings that maintain
consistency across different rendering views and clearly differentiate between
various objects and parts. These embeddings are transformed into a semantic
map, which directs a region-specific SDS process, enabling precise optimization
and compositional generation. By leveraging explicit semantic guidance, our
method unlocks the compositional capabilities of existing pre-trained diffusion
models, thereby achieving superior quality in 3D content generation,
particularly for complex objects and scenes. Experimental results demonstrate
that our SemanticSDS framework is highly effective for generating
state-of-the-art complex 3D content. Code:
https://github.com/YangLing0818/SemanticSDS-3D
A new Semantic Score Distillation Sampling approach improves accuracy and expressiveness in text-to-3D generation by using semantic embeddings and maps, enhancing pre-trained diffusion models.
Score Distillation Sampling Semantic Score Distillation Sampling semantic embeddings semantic map region-specific SDS pre-trained diffusion models