Fast Training of Diffusion Transformer with Extreme Masking for 3D Point
  Clouds Generation
Diffusion Transformers have recently shown remarkable effectiveness in
generating high-quality 3D point clouds. However, training voxel-based
diffusion models for high-resolution 3D voxels remains prohibitively expensive
due to the cubic complexity of attention operators, which arises from the
additional dimension of voxels. Motivated by the inherent redundancy of 3D
compared to 2D, we propose FastDiT-3D, a novel masked diffusion transformer
tailored for efficient 3D point cloud generation, which greatly reduces
training costs. Specifically, we draw inspiration from masked autoencoders to
dynamically operate the denoising process on masked voxelized point clouds. We
also propose a novel voxel-aware masking strategy to adaptively aggregate
background/foreground information from voxelized point clouds. Our method
achieves state-of-the-art performance with an extreme masking ratio of nearly
99%. Moreover, to improve multi-category 3D generation, we introduce
Mixture-of-Expert (MoE) in 3D diffusion model. Each category can learn a
distinct diffusion path with different experts, relieving gradient conflict.
Experimental results on the ShapeNet dataset demonstrate that our method
achieves state-of-the-art high-fidelity and diverse 3D point cloud generation
performance. Our FastDiT-3D improves 1-Nearest Neighbor Accuracy and Coverage
metrics when generating 128-resolution voxel point clouds, using only 6.5% of
the original training cost.
6579271b9999746238c9026c
Shentong Mo
admin_assigned
2023-12-13T09:30:00.174Z
631cbce0aa346997917f8faa
/avatars/abb996361d170b341429b98b9c91dbda.svg
Shentong
user
stoneMo
6579271b9999746238c9026d
Enze Xie
admin_assigned
2023-12-13T09:30:36.057Z
616961bdd3f656f79ad18ec1
https://cdn-avatars.huggingface.co/v1/production/uploads/616961bdd3f656f79ad18ec1/NYpxas378lDevRuUDepJJ.jpeg
xieenze
user
xieenze
6579271b9999746238c9026e
Yue Wu
6579271b9999746238c9026f
Junsong Chen
claimed_verified
2023-12-13T10:10:49.658Z
645b5b09bc7518912e1f9733
/avatars/4d35f728b41f93881a9b67c337f4d1df.svg
Chen
user
Lawrence-cj
6579271b9999746238c90270
Matthias Nießner
6579271b9999746238c90271
Zhenguo Li
Fast Training of Diffusion Transformer with Extreme Masking for 3D Point
  Clouds Generation
Diffusion Transformers have recently shown remarkable effectiveness in
generating high-quality 3D point clouds. However, training voxel-based
diffusion models for high-resolution 3D voxels remains prohibitively expensive
due to the cubic complexity of attention operators, which arises from the
additional dimension of voxels. Motivated by the inherent redundancy of 3D
compared to 2D, we propose FastDiT-3D, a novel masked diffusion transformer
tailored for efficient 3D point cloud generation, which greatly reduces
training costs. Specifically, we draw inspiration from masked autoencoders to
dynamically operate the denoising process on masked voxelized point clouds. We
also propose a novel voxel-aware masking strategy to adaptively aggregate
background/foreground information from voxelized point clouds. Our method
achieves state-of-the-art performance with an extreme masking ratio of nearly
99%. Moreover, to improve multi-category 3D generation, we introduce
Mixture-of-Expert (MoE) in 3D diffusion model. Each category can learn a
distinct diffusion path with different experts, relieving gradient conflict.
Experimental results on the ShapeNet dataset demonstrate that our method
achieves state-of-the-art high-fidelity and diverse 3D point cloud generation
performance. Our FastDiT-3D improves 1-Nearest Neighbor Accuracy and Coverage
metrics when generating 128-resolution voxel point clouds, using only 6.5% of
the original training cost.
FastDiT-3D, a masked diffusion transformer, efficiently generates high-quality 3D point clouds by reducing training costs through a novel voxel-aware masking strategy and integrating Mixture-of-Expert architecture.
Diffusion Transformers voxel-based diffusion models attention operators masked diffusion transformer masked autoencoders voxel-aware masking strategy Mixture-of-Expert ShapeNet 1-Nearest Neighbor Accuracy Coverage metrics