Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text
  Aligned Latent Representation
We present a novel alignment-before-generation approach to tackle the
challenging task of generating general 3D shapes based on 2D images or texts.
Directly learning a conditional generative model from images or texts to 3D
shapes is prone to producing inconsistent results with the conditions because
3D shapes have an additional dimension whose distribution significantly differs
from that of 2D images and texts. To bridge the domain gap among the three
modalities and facilitate multi-modal-conditioned 3D shape generation, we
explore representing 3D shapes in a shape-image-text-aligned space. Our
framework comprises two models: a Shape-Image-Text-Aligned Variational
Auto-Encoder (SITA-VAE) and a conditional Aligned Shape Latent Diffusion Model
(ASLDM). The former model encodes the 3D shapes into the shape latent space
aligned to the image and text and reconstructs the fine-grained 3D neural
fields corresponding to given shape embeddings via the transformer-based
decoder. The latter model learns a probabilistic mapping function from the
image or text space to the latent shape space. Our extensive experiments
demonstrate that our proposed approach can generate higher-quality and more
diverse 3D shapes that better semantically conform to the visual or textural
conditional inputs, validating the effectiveness of the
shape-image-text-aligned space for cross-modality 3D shape generation.
649e49ba8f9890dcc266e1b9
Zibo Zhao
649e49ba8f9890dcc266e1ba
Wen Liu
claimed_verified
2024-01-22T10:32:41.634Z
63198e8802fb322037332f2d
/avatars/d3f9c206e387df35beb0ed0ef1cdf865.svg
Wen Liu
user
doubility123
649e49ba8f9890dcc266e1bb
Xin Chen
claimed_verified
2023-12-20T08:21:46.538Z
63b543b2bda8d44adf2d7d0a
https://cdn-avatars.huggingface.co/v1/production/uploads/63b543b2bda8d44adf2d7d0a/G_AzO8iA-WwEcxvJEKVQ0.jpeg
Chen
user
ChrisChenXin
649e49ba8f9890dcc266e1bc
Xianfang Zeng
649e49ba8f9890dcc266e1bd
Rui Wang
claimed_verified
2023-12-12T08:09:22.865Z
630443483926de1f7ec6bda0
/avatars/e8e95ab0d03f58c61846ccc1318502fe.svg
Rui Wang
user
budui
649e49ba8f9890dcc266e1be
Pei Cheng
649e49ba8f9890dcc266e1bf
Bin Fu
649e49ba8f9890dcc266e1c0
Tao Chen
649e49ba8f9890dcc266e1c1
Gang Yu
claimed_verified
2023-06-30T10:23:20.237Z
63417332c5565a4b8d43a0d8
https://cdn-avatars.huggingface.co/v1/production/uploads/63417332c5565a4b8d43a0d8/MmnYG7Wu2Z_lqCBvTfJmy.png
Gang Yu
user
skicy
649e49ba8f9890dcc266e1c2
Shenghua Gao
Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text
  Aligned Latent Representation
We present a novel alignment-before-generation approach to tackle the
challenging task of generating general 3D shapes based on 2D images or texts.
Directly learning a conditional generative model from images or texts to 3D
shapes is prone to producing inconsistent results with the conditions because
3D shapes have an additional dimension whose distribution significantly differs
from that of 2D images and texts. To bridge the domain gap among the three
modalities and facilitate multi-modal-conditioned 3D shape generation, we
explore representing 3D shapes in a shape-image-text-aligned space. Our
framework comprises two models: a Shape-Image-Text-Aligned Variational
Auto-Encoder (SITA-VAE) and a conditional Aligned Shape Latent Diffusion Model
(ASLDM). The former model encodes the 3D shapes into the shape latent space
aligned to the image and text and reconstructs the fine-grained 3D neural
fields corresponding to given shape embeddings via the transformer-based
decoder. The latter model learns a probabilistic mapping function from the
image or text space to the latent shape space. Our extensive experiments
demonstrate that our proposed approach can generate higher-quality and more
diverse 3D shapes that better semantically conform to the visual or textural
conditional inputs, validating the effectiveness of the
shape-image-text-aligned space for cross-modality 3D shape generation.
A novel alignment-before-generation approach using SITA-VAE and ASLDM for generating high-quality 3D shapes from 2D images or texts leverages a shape-image-text-aligned space.
Shape-Image-Text-Aligned Variational Auto-Encoder (SITA-VAE) Aligned Shape Latent Diffusion Model (ASLDM) shape latent space transformer-based decoder semantic conforming