CLAY: A Controllable Large-scale Generative Model for Creating
  High-quality 3D Assets
In the realm of digital creativity, our potential to craft intricate 3D
worlds from imagination is often hampered by the limitations of existing
digital tools, which demand extensive expertise and efforts. To narrow this
disparity, we introduce CLAY, a 3D geometry and material generator designed to
effortlessly transform human imagination into intricate 3D digital structures.
CLAY supports classic text or image inputs as well as 3D-aware controls from
diverse primitives (multi-view images, voxels, bounding boxes, point clouds,
implicit representations, etc). At its core is a large-scale generative model
composed of a multi-resolution Variational Autoencoder (VAE) and a minimalistic
latent Diffusion Transformer (DiT), to extract rich 3D priors directly from a
diverse range of 3D geometries. Specifically, it adopts neural fields to
represent continuous and complete surfaces and uses a geometry generative
module with pure transformer blocks in latent space. We present a progressive
training scheme to train CLAY on an ultra large 3D model dataset obtained
through a carefully designed processing pipeline, resulting in a 3D native
geometry generator with 1.5 billion parameters. For appearance generation, CLAY
sets out to produce physically-based rendering (PBR) textures by employing a
multi-view material diffusion model that can generate 2K resolution textures
with diffuse, roughness, and metallic modalities. We demonstrate using CLAY for
a range of controllable 3D asset creations, from sketchy conceptual designs to
production ready assets with intricate details. Even first time users can
easily use CLAY to bring their vivid 3D imaginations to life, unleashing
unlimited creativity.
66750403b3bf319fd9608248
Longwen Zhang
claimed_verified
2024-07-26T07:41:05.873Z
636d12455aaed143cd665607
https://cdn-avatars.huggingface.co/v1/production/uploads/1679399015950-636d12455aaed143cd665607.png
ZLW
user
ZarkLngeW
66750403b3bf319fd9608249
Ziyu Wang
66750403b3bf319fd960824a
Qixuan Zhang
66750403b3bf319fd960824b
Qiwei Qiu
66750403b3bf319fd960824c
Anqi Pang
66750403b3bf319fd960824d
Haoran Jiang
66750403b3bf319fd960824e
Wei Yang
66750403b3bf319fd960824f
Lan Xu
66750403b3bf319fd9608250
Jingyi Yu
CLAY: A Controllable Large-scale Generative Model for Creating
  High-quality 3D Assets
In the realm of digital creativity, our potential to craft intricate 3D
worlds from imagination is often hampered by the limitations of existing
digital tools, which demand extensive expertise and efforts. To narrow this
disparity, we introduce CLAY, a 3D geometry and material generator designed to
effortlessly transform human imagination into intricate 3D digital structures.
CLAY supports classic text or image inputs as well as 3D-aware controls from
diverse primitives (multi-view images, voxels, bounding boxes, point clouds,
implicit representations, etc). At its core is a large-scale generative model
composed of a multi-resolution Variational Autoencoder (VAE) and a minimalistic
latent Diffusion Transformer (DiT), to extract rich 3D priors directly from a
diverse range of 3D geometries. Specifically, it adopts neural fields to
represent continuous and complete surfaces and uses a geometry generative
module with pure transformer blocks in latent space. We present a progressive
training scheme to train CLAY on an ultra large 3D model dataset obtained
through a carefully designed processing pipeline, resulting in a 3D native
geometry generator with 1.5 billion parameters. For appearance generation, CLAY
sets out to produce physically-based rendering (PBR) textures by employing a
multi-view material diffusion model that can generate 2K resolution textures
with diffuse, roughness, and metallic modalities. We demonstrate using CLAY for
a range of controllable 3D asset creations, from sketchy conceptual designs to
production ready assets with intricate details. Even first time users can
easily use CLAY to bring their vivid 3D imaginations to life, unleashing
unlimited creativity.
CLAY, a 3D geometry and material generator, uses a combination of VAE, DiT, and neural fields to produce high-quality 3D models and textures from various inputs, requiring minimal user expertise.
Multi-resolution Variational Autoencoder (VAE) latent Diffusion Transformer (DiT) neural fields geometry generative module progressive training ultra large 3D model dataset physically-based rendering (PBR) multi-view material diffusion model 2K resolution textures diffuse modalities roughness modalities metallic modalities