DreamScene360: Unconstrained Text-to-3D Scene Generation with Panoramic
  Gaussian Splatting
The increasing demand for virtual reality applications has highlighted the
significance of crafting immersive 3D assets. We present a text-to-3D
360^{circ} scene generation pipeline that facilitates the creation of
comprehensive 360^{circ} scenes for in-the-wild environments in a matter of
minutes. Our approach utilizes the generative power of a 2D diffusion model and
prompt self-refinement to create a high-quality and globally coherent panoramic
image. This image acts as a preliminary "flat" (2D) scene representation.
Subsequently, it is lifted into 3D Gaussians, employing splatting techniques to
enable real-time exploration. To produce consistent 3D geometry, our pipeline
constructs a spatially coherent structure by aligning the 2D monocular depth
into a globally optimized point cloud. This point cloud serves as the initial
state for the centroids of 3D Gaussians. In order to address invisible issues
inherent in single-view inputs, we impose semantic and geometric constraints on
both synthesized and input camera views as regularizations. These guide the
optimization of Gaussians, aiding in the reconstruction of unseen regions. In
summary, our method offers a globally consistent 3D scene within a
360^{circ} perspective, providing an enhanced immersive experience over
existing techniques. Project website at: http://dreamscene360.github.io/
661743070795b7cd10a2ade6
Shijie Zhou
claimed_verified
2024-04-11T06:42:05.581Z
642a276516d4d8293c9a47e8
/avatars/80e6db8bc2544f3486b11b57858a8692.svg
Shijie Zhou
user
shijiezhou
661743070795b7cd10a2ade7
Zhiwen Fan
admin_assigned
2024-04-18T13:24:16.216Z
6526386e1c6a09292d8d0a22
/avatars/471de830de2d775d35368678c1579f87.svg
fan
user
Fanzhiwen
661743070795b7cd10a2ade8
Dejia Xu
admin_assigned
2024-04-18T13:24:25.149Z
62f687a0c58915315c4ff75d
/avatars/b657180c7666735062782edd4f6a69c9.svg
Dejia Xu
user
ir1d
661743070795b7cd10a2ade9
Haoran Chang
661743070795b7cd10a2adea
Pradyumna Chari
admin_assigned
2024-04-18T13:24:42.880Z
64e2ac52cf8c236f30176499
/avatars/0438be7e7b502a2892444249b49b92fe.svg
Pradyumna Chari
user
pradyumnachari
661743070795b7cd10a2adeb
Tejas Bharadwaj
661743070795b7cd10a2adec
Suya You
661743070795b7cd10a2aded
Zhangyang Wang
661743070795b7cd10a2adee
Achuta Kadambi
DreamScene360: Unconstrained Text-to-3D Scene Generation with Panoramic
  Gaussian Splatting
The increasing demand for virtual reality applications has highlighted the
significance of crafting immersive 3D assets. We present a text-to-3D
360^{circ} scene generation pipeline that facilitates the creation of
comprehensive 360^{circ} scenes for in-the-wild environments in a matter of
minutes. Our approach utilizes the generative power of a 2D diffusion model and
prompt self-refinement to create a high-quality and globally coherent panoramic
image. This image acts as a preliminary "flat" (2D) scene representation.
Subsequently, it is lifted into 3D Gaussians, employing splatting techniques to
enable real-time exploration. To produce consistent 3D geometry, our pipeline
constructs a spatially coherent structure by aligning the 2D monocular depth
into a globally optimized point cloud. This point cloud serves as the initial
state for the centroids of 3D Gaussians. In order to address invisible issues
inherent in single-view inputs, we impose semantic and geometric constraints on
both synthesized and input camera views as regularizations. These guide the
optimization of Gaussians, aiding in the reconstruction of unseen regions. In
summary, our method offers a globally consistent 3D scene within a
360^{circ} perspective, providing an enhanced immersive experience over
existing techniques. Project website at: http://dreamscene360.github.io/
A text-to-3D 360° scene generation pipeline creates immersive environments using a 2D diffusion model and lifts flat panoramas into 3D Gaussians with semantic and geometric constraints.
diffusion model prompt self-refinement panoramic image 3D Gaussians splatting techniques 2D monocular depth point cloud semantic constraints geometric constraints