LangSplat: 3D Language Gaussian Splatting
Human lives in a 3D world and commonly uses natural language to interact with
a 3D scene. Modeling a 3D language field to support open-ended language queries
in 3D has gained increasing attention recently. This paper introduces
LangSplat, which constructs a 3D language field that enables precise and
efficient open-vocabulary querying within 3D spaces. Unlike existing methods
that ground CLIP language embeddings in a NeRF model, LangSplat advances the
field by utilizing a collection of 3D Gaussians, each encoding language
features distilled from CLIP, to represent the language field. By employing a
tile-based splatting technique for rendering language features, we circumvent
the costly rendering process inherent in NeRF. Instead of directly learning
CLIP embeddings, LangSplat first trains a scene-wise language autoencoder and
then learns language features on the scene-specific latent space, thereby
alleviating substantial memory demands imposed by explicit modeling. Existing
methods struggle with imprecise and vague 3D language fields, which fail to
discern clear boundaries between objects. We delve into this issue and propose
to learn hierarchical semantics using SAM, thereby eliminating the need for
extensively querying the language field across various scales and the
regularization of DINO features. Extensive experiments on open-vocabulary 3D
object localization and semantic segmentation demonstrate that LangSplat
significantly outperforms the previous state-of-the-art method LERF by a large
margin. Notably, LangSplat is extremely efficient, achieving a {\speed}
times speedup compared to LERF at the resolution of 1440 times 1080. We
strongly recommend readers to check out our video results at
https://langsplat.github.io
658bba53ddfe71efe7db540f
Minghan Qin
claimed_verified
2023-12-29T09:00:41.663Z
642002b51ccd411979d72b18
https://cdn-avatars.huggingface.co/v1/production/uploads/642002b51ccd411979d72b18/JO9e0o8fAFNC-eYFbx_JW.png
Minghan Qin
user
Qmh
658bba53ddfe71efe7db5410
Wanhua Li
claimed_verified
2023-12-27T11:05:46.069Z
658bb7e47459b6e471b9d2e6
/avatars/efd8051b468b4dbcb5d149479de67c58.svg
Wanhua Li
user
EthanTaylor
658bba53ddfe71efe7db5411
Jiawei Zhou
658bba53ddfe71efe7db5412
Haoqian Wang
658bba53ddfe71efe7db5413
Hanspeter Pfister
LangSplat: 3D Language Gaussian Splatting
Human lives in a 3D world and commonly uses natural language to interact with
a 3D scene. Modeling a 3D language field to support open-ended language queries
in 3D has gained increasing attention recently. This paper introduces
LangSplat, which constructs a 3D language field that enables precise and
efficient open-vocabulary querying within 3D spaces. Unlike existing methods
that ground CLIP language embeddings in a NeRF model, LangSplat advances the
field by utilizing a collection of 3D Gaussians, each encoding language
features distilled from CLIP, to represent the language field. By employing a
tile-based splatting technique for rendering language features, we circumvent
the costly rendering process inherent in NeRF. Instead of directly learning
CLIP embeddings, LangSplat first trains a scene-wise language autoencoder and
then learns language features on the scene-specific latent space, thereby
alleviating substantial memory demands imposed by explicit modeling. Existing
methods struggle with imprecise and vague 3D language fields, which fail to
discern clear boundaries between objects. We delve into this issue and propose
to learn hierarchical semantics using SAM, thereby eliminating the need for
extensively querying the language field across various scales and the
regularization of DINO features. Extensive experiments on open-vocabulary 3D
object localization and semantic segmentation demonstrate that LangSplat
significantly outperforms the previous state-of-the-art method LERF by a large
margin. Notably, LangSplat is extremely efficient, achieving a {\speed}
times speedup compared to LERF at the resolution of 1440 times 1080. We
strongly recommend readers to check out our video results at
https://langsplat.github.io
LangSplat constructs a 3D language field using a splatting technique with hierarchical semantics, enabling precise and efficient open-vocabulary querying in 3D spaces while significantly outperforming existing methods.
CLIP NeRF 3D Gaussians tile-based splatting scene-wise language autoencoder SAM hierarchical semantics DINO features open-vocabulary 3D object localization semantic segmentation LERF