Mosaic-SDF for 3D Generative Models
Current diffusion or flow-based generative models for 3D shapes divide to
two: distilling pre-trained 2D image diffusion models, and training directly on
3D shapes. When training a diffusion or flow models on 3D shapes a crucial
design choice is the shape representation. An effective shape representation
needs to adhere three design principles: it should allow an efficient
conversion of large 3D datasets to the representation form; it should provide a
good tradeoff of approximation power versus number of parameters; and it should
have a simple tensorial form that is compatible with existing powerful neural
architectures. While standard 3D shape representations such as volumetric grids
and point clouds do not adhere to all these principles simultaneously, we
advocate in this paper a new representation that does. We introduce Mosaic-SDF
(M-SDF): a simple 3D shape representation that approximates the Signed Distance
Function (SDF) of a given shape by using a set of local grids spread near the
shape's boundary. The M-SDF representation is fast to compute for each shape
individually making it readily parallelizable; it is parameter efficient as it
only covers the space around the shape's boundary; and it has a simple matrix
form, compatible with Transformer-based architectures. We demonstrate the
efficacy of the M-SDF representation by using it to train a 3D generative flow
model including class-conditioned generation with the 3D Warehouse dataset, and
text-to-3D generation using a dataset of about 600k caption-shape pairs.
657bd80564dab2f4c9fcf770
Lior Yariv
admin_assigned
2023-12-15T11:10:16.637Z
63f351070be81bdc5d90fd61
/avatars/752d8bc512cdad704bc814876514bc87.svg
Lior Yariv
user
lioryariv
657bd80564dab2f4c9fcf771
Omri Puny
657bd80564dab2f4c9fcf772
Natalia Neverova
657bd80564dab2f4c9fcf773
Oran Gafni
admin_assigned
2023-12-15T11:09:42.274Z
6345a97d547c70e4b7ce9169
/avatars/5d233e65eadfa7d9ef0390a584df5451.svg
Oran Gafni
user
Oran22
657bd80564dab2f4c9fcf774
Yaron Lipman
Mosaic-SDF for 3D Generative Models
Current diffusion or flow-based generative models for 3D shapes divide to
two: distilling pre-trained 2D image diffusion models, and training directly on
3D shapes. When training a diffusion or flow models on 3D shapes a crucial
design choice is the shape representation. An effective shape representation
needs to adhere three design principles: it should allow an efficient
conversion of large 3D datasets to the representation form; it should provide a
good tradeoff of approximation power versus number of parameters; and it should
have a simple tensorial form that is compatible with existing powerful neural
architectures. While standard 3D shape representations such as volumetric grids
and point clouds do not adhere to all these principles simultaneously, we
advocate in this paper a new representation that does. We introduce Mosaic-SDF
(M-SDF): a simple 3D shape representation that approximates the Signed Distance
Function (SDF) of a given shape by using a set of local grids spread near the
shape's boundary. The M-SDF representation is fast to compute for each shape
individually making it readily parallelizable; it is parameter efficient as it
only covers the space around the shape's boundary; and it has a simple matrix
form, compatible with Transformer-based architectures. We demonstrate the
efficacy of the M-SDF representation by using it to train a 3D generative flow
model including class-conditioned generation with the 3D Warehouse dataset, and
text-to-3D generation using a dataset of about 600k caption-shape pairs.
A new Mosaic-SDF representation for 3D shapes enables efficient, parameter-efficient, and parallelizable generation of 3D shapes using generative flow models.
diffusion models flow-based generative models 3D shapes shape representation volumetric grids point clouds Signed Distance Function (SDF) Mosaic-SDF Transformer-based architectures class-conditioned generation text-to-3D generation