Urban Architect: Steerable 3D Urban Scene Generation with Layout Prior
Text-to-3D generation has achieved remarkable success via large-scale
text-to-image diffusion models. Nevertheless, there is no paradigm for scaling
up the methodology to urban scale. Urban scenes, characterized by numerous
elements, intricate arrangement relationships, and vast scale, present a
formidable barrier to the interpretability of ambiguous textual descriptions
for effective model optimization. In this work, we surmount the limitations by
introducing a compositional 3D layout representation into text-to-3D paradigm,
serving as an additional prior. It comprises a set of semantic primitives with
simple geometric structures and explicit arrangement relationships,
complementing textual descriptions and enabling steerable generation. Upon
this, we propose two modifications -- (1) We introduce Layout-Guided
Variational Score Distillation to address model optimization inadequacies. It
conditions the score distillation sampling process with geometric and semantic
constraints of 3D layouts. (2) To handle the unbounded nature of urban scenes,
we represent 3D scene with a Scalable Hash Grid structure, incrementally
adapting to the growing scale of urban scenes. Extensive experiments
substantiate the capability of our framework to scale text-to-3D generation to
large-scale urban scenes that cover over 1000m driving distance for the first
time. We also present various scene editing demonstrations, showing the powers
of steerable urban scene generation. Website: https://urbanarchitect.github.io.
66173c72843718c34d0d9983
Fan Lu
claimed_verified
2024-04-19T07:39:10.807Z
6429759de5f33939cf3a57a6
/avatars/16c40c87802ed3463a64174e4a66149b.svg
Fan Lu
user
FanLu97
66173c72843718c34d0d9984
Kwan-Yee Lin
admin_assigned
2024-04-18T14:26:22.847Z
64b9226b25e5f8a3e0fc1bda
https://cdn-avatars.huggingface.co/v1/production/uploads/64b9226b25e5f8a3e0fc1bda/sU5ZN_a3VeTela--6sX5-.jpeg
Kwan-Yee Lin
user
KwanYee
66173c72843718c34d0d9985
Yan Xu
66173c72843718c34d0d9986
Hongsheng Li
admin_assigned
2024-04-18T14:26:49.874Z
65c04e9c27a5fdca81abcbd9
/avatars/12a155683c824fa23da4a9e2bed4f64e.svg
Hongsheng LI
user
hsli-cuhk
66173c72843718c34d0d9987
Guang Chen
66173c72843718c34d0d9988
Changjun Jiang
Urban Architect: Steerable 3D Urban Scene Generation with Layout Prior
Text-to-3D generation has achieved remarkable success via large-scale
text-to-image diffusion models. Nevertheless, there is no paradigm for scaling
up the methodology to urban scale. Urban scenes, characterized by numerous
elements, intricate arrangement relationships, and vast scale, present a
formidable barrier to the interpretability of ambiguous textual descriptions
for effective model optimization. In this work, we surmount the limitations by
introducing a compositional 3D layout representation into text-to-3D paradigm,
serving as an additional prior. It comprises a set of semantic primitives with
simple geometric structures and explicit arrangement relationships,
complementing textual descriptions and enabling steerable generation. Upon
this, we propose two modifications -- (1) We introduce Layout-Guided
Variational Score Distillation to address model optimization inadequacies. It
conditions the score distillation sampling process with geometric and semantic
constraints of 3D layouts. (2) To handle the unbounded nature of urban scenes,
we represent 3D scene with a Scalable Hash Grid structure, incrementally
adapting to the growing scale of urban scenes. Extensive experiments
substantiate the capability of our framework to scale text-to-3D generation to
large-scale urban scenes that cover over 1000m driving distance for the first
time. We also present various scene editing demonstrations, showing the powers
of steerable urban scene generation. Website: https://urbanarchitect.github.io.
Introducing a compositional 3D layout representation and modifications to text-to-3D generation techniques enables scaling to large-scale urban scenes with enhanced interpretability and steerable generation.
text-to-3D generation text-to-image diffusion models compositional 3D layout representation semantic primitives geometric structures Layout-Guided Variational Score Distillation score distillation geometric constraints semantic constraints Scalable Hash Grid steerable generation