Sampling 3D Gaussian Scenes in Seconds with Latent Diffusion Models
We present a latent diffusion model over 3D scenes, that can be trained using
only 2D image data. To achieve this, we first design an autoencoder that maps
multi-view images to 3D Gaussian splats, and simultaneously builds a compressed
latent representation of these splats. Then, we train a multi-view diffusion
model over the latent space to learn an efficient generative model. This
pipeline does not require object masks nor depths, and is suitable for complex
scenes with arbitrary camera positions. We conduct careful experiments on two
large-scale datasets of complex real-world scenes -- MVImgNet and
RealEstate10K. We show that our approach enables generating 3D scenes in as
little as 0.2 seconds, either from scratch, from a single input view, or from
sparse input views. It produces diverse and high-quality results while running
an order of magnitude faster than non-latent diffusion models and earlier
NeRF-based generative models
667609215eaa9dd299f4feee
Paul Henderson
claimed_verified
2024-06-24T07:30:00.485Z
66749366b82e48404d157a66
/avatars/a56066447b0ebcc26c31ed2388a08707.svg
Paul Henderson
user
pmh47
667609215eaa9dd299f4feef
Melonie de Almeida
667609215eaa9dd299f4fef0
Daniela Ivanova
667609215eaa9dd299f4fef1
Titas Anciukevičius
Sampling 3D Gaussian Scenes in Seconds with Latent Diffusion Models
We present a latent diffusion model over 3D scenes, that can be trained using
only 2D image data. To achieve this, we first design an autoencoder that maps
multi-view images to 3D Gaussian splats, and simultaneously builds a compressed
latent representation of these splats. Then, we train a multi-view diffusion
model over the latent space to learn an efficient generative model. This
pipeline does not require object masks nor depths, and is suitable for complex
scenes with arbitrary camera positions. We conduct careful experiments on two
large-scale datasets of complex real-world scenes -- MVImgNet and
RealEstate10K. We show that our approach enables generating 3D scenes in as
little as 0.2 seconds, either from scratch, from a single input view, or from
sparse input views. It produces diverse and high-quality results while running
an order of magnitude faster than non-latent diffusion models and earlier
NeRF-based generative models
A latent diffusion model capable of generating 3D scenes from 2D images enables fast and high-quality scene synthesis without masks or depth data.
latent diffusion model multi-view images 3D Gaussian splats compressed latent representation multi-view diffusion model latent space MVImgNet RealEstate10K scene synthesis generative model NeRF-based generative models