Imaginarium: Vision-guided High-Quality 3D Scene Layout Generation
Generating artistic and coherent 3D scene layouts is crucial in digital
content creation. Traditional optimization-based methods are often constrained
by cumbersome manual rules, while deep generative models face challenges in
producing content with richness and diversity. Furthermore, approaches that
utilize large language models frequently lack robustness and fail to accurately
capture complex spatial relationships. To address these challenges, this paper
presents a novel vision-guided 3D layout generation system. We first construct
a high-quality asset library containing 2,037 scene assets and 147 3D scene
layouts. Subsequently, we employ an image generation model to expand prompt
representations into images, fine-tuning it to align with our asset library. We
then develop a robust image parsing module to recover the 3D layout of scenes
based on visual semantics and geometric information. Finally, we optimize the
scene layout using scene graphs and overall visual semantics to ensure logical
coherence and alignment with the images. Extensive user testing demonstrates
that our algorithm significantly outperforms existing methods in terms of
layout richness and quality. The code and dataset will be available at
https://github.com/HiHiAllen/Imaginarium.
68f593d08589920bf4d320de
Xiaoming Zhu
68f593d08589920bf4d320df
Xu Huang
68f593d08589920bf4d320e0
Qinghongbing Xie
68f593d08589920bf4d320e1
Zhi Deng
68f593d08589920bf4d320e2
Junsheng Yu
68f593d08589920bf4d320e3
Yirui Guan
68f593d08589920bf4d320e4
Zhongyuan Liu
68f593d08589920bf4d320e5
Lin Zhu
68f593d08589920bf4d320e6
Qijun Zhao
68f593d08589920bf4d320e7
Ligang Liu
68f593d08589920bf4d320e8
Long Zeng
Imaginarium: Vision-guided High-Quality 3D Scene Layout Generation
Generating artistic and coherent 3D scene layouts is crucial in digital
content creation. Traditional optimization-based methods are often constrained
by cumbersome manual rules, while deep generative models face challenges in
producing content with richness and diversity. Furthermore, approaches that
utilize large language models frequently lack robustness and fail to accurately
capture complex spatial relationships. To address these challenges, this paper
presents a novel vision-guided 3D layout generation system. We first construct
a high-quality asset library containing 2,037 scene assets and 147 3D scene
layouts. Subsequently, we employ an image generation model to expand prompt
representations into images, fine-tuning it to align with our asset library. We
then develop a robust image parsing module to recover the 3D layout of scenes
based on visual semantics and geometric information. Finally, we optimize the
scene layout using scene graphs and overall visual semantics to ensure logical
coherence and alignment with the images. Extensive user testing demonstrates
that our algorithm significantly outperforms existing methods in terms of
layout richness and quality. The code and dataset will be available at
https://github.com/HiHiAllen/Imaginarium.
A vision-guided 3D layout generation system uses an image generation model and scene graphs to produce rich and coherent 3D scenes from prompts.
image generation model image parsing module scene graphs 3D layout generation visual semantics geometric information scene graphs logical coherence