GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphs
As pretrained text-to-image diffusion models become increasingly powerful,
recent efforts have been made to distill knowledge from these text-to-image
pretrained models for optimizing a text-guided 3D model. Most of the existing
methods generate a holistic 3D model from a plain text input. This can be
problematic when the text describes a complex scene with multiple objects,
because the vectorized text embeddings are inherently unable to capture a
complex description with multiple entities and relationships. Holistic 3D
modeling of the entire scene further prevents accurate grounding of text
entities and concepts. To address this limitation, we propose GraphDreamer, a
novel framework to generate compositional 3D scenes from scene graphs, where
objects are represented as nodes and their interactions as edges. By exploiting
node and edge information in scene graphs, our method makes better use of the
pretrained text-to-image diffusion model and is able to fully disentangle
different objects without image-level supervision. To facilitate modeling of
object-wise relationships, we use signed distance fields as representation and
impose a constraint to avoid inter-penetration of objects. To avoid manual
scene graph creation, we design a text prompt for ChatGPT to generate scene
graphs based on text inputs. We conduct both qualitative and quantitative
experiments to validate the effectiveness of GraphDreamer in generating
high-fidelity compositional 3D scenes with disentangled object entities.
656d30f1f7be0986b4b5a351
Gege Gao
admin_assigned
2023-12-04T09:46:08.169Z
6476473442e5f5297459e8f1
/avatars/3fe8549dfd05a167d5b7b0287d5a7d46.svg
Gege Gao
user
GGGHSL
656d30f1f7be0986b4b5a352
Weiyang Liu
admin_assigned
2023-12-04T09:39:15.183Z
648905d1a15c43c791d4381f
https://cdn-avatars.huggingface.co/v1/production/uploads/648905d1a15c43c791d4381f/GpqGBzsLiMHX0gWZEz3qn.jpeg
Weiyang Liu
user
wy1iu
656d30f1f7be0986b4b5a353
Anpei Chen
admin_assigned
2023-12-04T09:39:30.232Z
63ea4b4be7f148f46015099f
https://cdn-avatars.huggingface.co/v1/production/uploads/63ea4b4be7f148f46015099f/tTuZYvtmM2IIGBH8BpKA1.jpeg
ANPEI CHEN
user
apchen
656d30f1f7be0986b4b5a354
Andreas Geiger
admin_assigned
2023-12-04T09:39:37.165Z
620cae049086f3c07f01e3d5
https://cdn-avatars.huggingface.co/v1/production/uploads/1644998139538-noauth.jpeg
Andreas Geiger
user
andreas-geiger
656d30f1f7be0986b4b5a355
Bernhard Schölkopf
GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphs
As pretrained text-to-image diffusion models become increasingly powerful,
recent efforts have been made to distill knowledge from these text-to-image
pretrained models for optimizing a text-guided 3D model. Most of the existing
methods generate a holistic 3D model from a plain text input. This can be
problematic when the text describes a complex scene with multiple objects,
because the vectorized text embeddings are inherently unable to capture a
complex description with multiple entities and relationships. Holistic 3D
modeling of the entire scene further prevents accurate grounding of text
entities and concepts. To address this limitation, we propose GraphDreamer, a
novel framework to generate compositional 3D scenes from scene graphs, where
objects are represented as nodes and their interactions as edges. By exploiting
node and edge information in scene graphs, our method makes better use of the
pretrained text-to-image diffusion model and is able to fully disentangle
different objects without image-level supervision. To facilitate modeling of
object-wise relationships, we use signed distance fields as representation and
impose a constraint to avoid inter-penetration of objects. To avoid manual
scene graph creation, we design a text prompt for ChatGPT to generate scene
graphs based on text inputs. We conduct both qualitative and quantitative
experiments to validate the effectiveness of GraphDreamer in generating
high-fidelity compositional 3D scenes with disentangled object entities.
GraphDreamer generates complex 3D scenes from text inputs by using scene graphs to disentangle object entities and relationships, leveraging pretrained text-to-image diffusion models.
pretrained text-to-image diffusion models scene graphs nodes edges text-to-image diffusion model signed distance fields inter-penetration ChatGPT