How Many Tokens Do 3D Point Cloud Transformer Architectures Really Need?
Tuan Tran, Duy M. H. Nguyen, Hoai-Chau Tran, Michael Barz, Khoa D Doan, Roger Wattenhofer, Vien Ngo, Mathias Niepert, Daniel Sonntag, Paul Swoboda
In recent years, 3D point cloud transformers have emerged as powerful tools for various tasks such as semantic segmentation, object detection, and reconstruction. While these models have demonstrated remarkable performance, they are often computationally expensive due to the large number of tokens required to process 3D point clouds. Surprisingly, we observe that despite the high computational cost of these models, they actually require only a small fraction of the tokens to achieve their performance. This discovery challenges the conventional assumption that using more tokens directly improves performance. In this work, we demonstrate that with a proper token merging strategy for 3D point cloud transformers, estimating the importance of voxel tokens based on their spatial structures, we are able to reduce the token usage by up to 95-98\% with minimal performance degradation. Our findings highlight that many existing state-of-the-art models are unnecessarily token-heavy, and our method provides a more efficient way to handle tokenization without compromising accuracy. This phenomenon holds across multiple domains, including 3D semantic segmentation, point cloud reconstruction, and object detection, demonstrating the potential for scalability and efficiency in 3D vision tasks.