Compress3D: a Compressed Latent Space for 3D Generation from a Single
  Image
3D generation has witnessed significant advancements, yet efficiently
producing high-quality 3D assets from a single image remains challenging. In
this paper, we present a triplane autoencoder, which encodes 3D models into a
compact triplane latent space to effectively compress both the 3D geometry and
texture information. Within the autoencoder framework, we introduce a 3D-aware
cross-attention mechanism, which utilizes low-resolution latent representations
to query features from a high-resolution 3D feature volume, thereby enhancing
the representation capacity of the latent space. Subsequently, we train a
diffusion model on this refined latent space. In contrast to solely relying on
image embedding for 3D generation, our proposed method advocates for the
simultaneous utilization of both image embedding and shape embedding as
conditions. Specifically, the shape embedding is estimated via a diffusion
prior model conditioned on the image embedding. Through comprehensive
experiments, we demonstrate that our method outperforms state-of-the-art
algorithms, achieving superior performance while requiring less training data
and time. Our approach enables the generation of high-quality 3D assets in
merely 7 seconds on a single A100 GPU.
65fba5f7dfd0ff86aa77dd44
Bowen Zhang
65fba5f7dfd0ff86aa77dd45
Tianyu Yang
65fba5f7dfd0ff86aa77dd46
Yu Li
65fba5f7dfd0ff86aa77dd47
Lei Zhang
65fba5f7dfd0ff86aa77dd48
Xi Zhao
Compress3D: a Compressed Latent Space for 3D Generation from a Single
  Image
3D generation has witnessed significant advancements, yet efficiently
producing high-quality 3D assets from a single image remains challenging. In
this paper, we present a triplane autoencoder, which encodes 3D models into a
compact triplane latent space to effectively compress both the 3D geometry and
texture information. Within the autoencoder framework, we introduce a 3D-aware
cross-attention mechanism, which utilizes low-resolution latent representations
to query features from a high-resolution 3D feature volume, thereby enhancing
the representation capacity of the latent space. Subsequently, we train a
diffusion model on this refined latent space. In contrast to solely relying on
image embedding for 3D generation, our proposed method advocates for the
simultaneous utilization of both image embedding and shape embedding as
conditions. Specifically, the shape embedding is estimated via a diffusion
prior model conditioned on the image embedding. Through comprehensive
experiments, we demonstrate that our method outperforms state-of-the-art
algorithms, achieving superior performance while requiring less training data
and time. Our approach enables the generation of high-quality 3D assets in
merely 7 seconds on a single A100 GPU.
A triplane autoencoder with a 3D-aware cross-attention mechanism and diffusion model outperforms existing methods in generating high-quality 3D assets from a single image, using both image and shape embeddings.
triplane autoencoder triplane latent space 3D-aware cross-attention mechanism high-resolution 3D feature volume diffusion model image embedding shape embedding diffusion prior model A100 GPU