High-dimensional 3D Language Gaussian Splatting with 450+ FPS
Wanhua Li, Yujie Zhao, Minghan Qin, Yang Liu, Yuanhao Cai, Chuang Gan, Hanspeter Pfister
In this paper, we introduce LangSplatV2, which achieves high-dimensional featuresplatting at 476.2 FPS and 3D open-vocabulary text querying at 384.6 FPS forhigh-resolution images, providing a 42 × speedup and a 47 × boost over LangSplatrespectively, along with improved query accuracy. LangSplat employs GaussianSplatting to embed 2D CLIP language features into 3D, significantly enhancingspeed and learning a precise 3D language field with SAM semantics. Such advancements in 3D language fields are crucial for applications that require languageinteraction within complex scenes. However, LangSplat does not yet achieve real-time performance (8.2 FPS), even with advanced A100 GPUs, severely limitingits broader application. In this paper, we first conduct a detailed time analysis ofLangSplat, identifying the heavyweight decoder as the primary speed bottleneck.Our solution, LangSplatV2 assumes that each Gaussian acts as a sparse code withina global dictionary, leading to the learning of a 3D sparse coefficient field thatentirely eliminates the need for a heavyweight decoder. By leveraging this sparsity,we further propose an efficient sparse coefficient splatting method with CUDA optimization, rendering high-dimensional feature maps at high quality while incurringonly the time cost of splatting an ultra-low-dimensional feature. Our experimentalresults demonstrate that LangSplatV2 not only achieves better or competitive queryaccuracy but is also significantly faster.