DiT-3D: Exploring Plain Diffusion Transformers for 3D Shape Generation
Recent Diffusion Transformers (e.g., DiT) have demonstrated their powerful
effectiveness in generating high-quality 2D images. However, it is still being
determined whether the Transformer architecture performs equally well in 3D
shape generation, as previous 3D diffusion methods mostly adopted the U-Net
architecture. To bridge this gap, we propose a novel Diffusion Transformer for
3D shape generation, namely DiT-3D, which can directly operate the denoising
process on voxelized point clouds using plain Transformers. Compared to
existing U-Net approaches, our DiT-3D is more scalable in model size and
produces much higher quality generations. Specifically, the DiT-3D adopts the
design philosophy of DiT but modifies it by incorporating 3D positional and
patch embeddings to adaptively aggregate input from voxelized point clouds. To
reduce the computational cost of self-attention in 3D shape generation, we
incorporate 3D window attention into Transformer blocks, as the increased 3D
token length resulting from the additional dimension of voxels can lead to high
computation. Finally, linear and devoxelization layers are used to predict the
denoised point clouds. In addition, our transformer architecture supports
efficient fine-tuning from 2D to 3D, where the pre-trained DiT-2D checkpoint on
ImageNet can significantly improve DiT-3D on ShapeNet. Experimental results on
the ShapeNet dataset demonstrate that the proposed DiT-3D achieves
state-of-the-art performance in high-fidelity and diverse 3D point cloud
generation. In particular, our DiT-3D decreases the 1-Nearest Neighbor Accuracy
of the state-of-the-art method by 4.59 and increases the Coverage metric by
3.51 when evaluated on Chamfer Distance.
64a62fb898a3e01fd6929da0
Shentong Mo
claimed_verified
2023-07-07T07:35:12.294Z
631cbce0aa346997917f8faa
/avatars/abb996361d170b341429b98b9c91dbda.svg
Shentong
user
stoneMo
64a62fb898a3e01fd6929da1
Enze Xie
64a62fb898a3e01fd6929da2
Ruihang Chu
claimed_verified
2025-07-18T07:51:28.113Z
642e3bcb958faf258a40e89c
/avatars/213501def37dc53032cee17e37fcc4c1.svg
Ruihang Chu
user
Ruihang
64a62fb898a3e01fd6929da3
Lewei Yao
64a62fb898a3e01fd6929da4
Lanqing Hong
64a62fb898a3e01fd6929da5
Matthias Nießner
64a62fb898a3e01fd6929da6
Zhenguo Li
DiT-3D: Exploring Plain Diffusion Transformers for 3D Shape Generation
Recent Diffusion Transformers (e.g., DiT) have demonstrated their powerful
effectiveness in generating high-quality 2D images. However, it is still being
determined whether the Transformer architecture performs equally well in 3D
shape generation, as previous 3D diffusion methods mostly adopted the U-Net
architecture. To bridge this gap, we propose a novel Diffusion Transformer for
3D shape generation, namely DiT-3D, which can directly operate the denoising
process on voxelized point clouds using plain Transformers. Compared to
existing U-Net approaches, our DiT-3D is more scalable in model size and
produces much higher quality generations. Specifically, the DiT-3D adopts the
design philosophy of DiT but modifies it by incorporating 3D positional and
patch embeddings to adaptively aggregate input from voxelized point clouds. To
reduce the computational cost of self-attention in 3D shape generation, we
incorporate 3D window attention into Transformer blocks, as the increased 3D
token length resulting from the additional dimension of voxels can lead to high
computation. Finally, linear and devoxelization layers are used to predict the
denoised point clouds. In addition, our transformer architecture supports
efficient fine-tuning from 2D to 3D, where the pre-trained DiT-2D checkpoint on
ImageNet can significantly improve DiT-3D on ShapeNet. Experimental results on
the ShapeNet dataset demonstrate that the proposed DiT-3D achieves
state-of-the-art performance in high-fidelity and diverse 3D point cloud
generation. In particular, our DiT-3D decreases the 1-Nearest Neighbor Accuracy
of the state-of-the-art method by 4.59 and increases the Coverage metric by
3.51 when evaluated on Chamfer Distance.
A novel Diffusion Transformer architecture for 3D shape generation, DiT-3D, using voxelized point clouds achieves superior performance compared to U-Net methods, employing 3D positional embeddings and 3D window attention to reduce computational costs.
Diffusion Transformers DiT DiT-3D voxelized point clouds U-Net 3D positional embeddings patch embeddings 3D window attention linear layers devoxelization layers efficient fine-tuning ShapeNet 1-Nearest Neighbor Accuracy Coverage metric Chamfer Distance