4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large
  Language Models
Learning 4D language fields to enable time-sensitive, open-ended language
queries in dynamic scenes is essential for many real-world applications. While
LangSplat successfully grounds CLIP features into 3D Gaussian representations,
achieving precision and efficiency in 3D static scenes, it lacks the ability to
handle dynamic 4D fields as CLIP, designed for static image-text tasks, cannot
capture temporal dynamics in videos. Real-world environments are inherently
dynamic, with object semantics evolving over time. Building a precise 4D
language field necessitates obtaining pixel-aligned, object-wise video
features, which current vision models struggle to achieve. To address these
challenges, we propose 4D LangSplat, which learns 4D language fields to handle
time-agnostic or time-sensitive open-vocabulary queries in dynamic scenes
efficiently. 4D LangSplat bypasses learning the language field from vision
features and instead learns directly from text generated from object-wise video
captions via Multimodal Large Language Models (MLLMs). Specifically, we propose
a multimodal object-wise video prompting method, consisting of visual and text
prompts that guide MLLMs to generate detailed, temporally consistent,
high-quality captions for objects throughout a video. These captions are
encoded using a Large Language Model into high-quality sentence embeddings,
which then serve as pixel-aligned, object-specific feature supervision,
facilitating open-vocabulary text queries through shared embedding spaces.
Recognizing that objects in 4D scenes exhibit smooth transitions across states,
we further propose a status deformable network to model these continuous
changes over time effectively. Our results across multiple benchmarks
demonstrate that 4D LangSplat attains precise and efficient results for both
time-sensitive and time-agnostic open-vocabulary queries.
67d3adf57360ea908cf5f0bc
Wanhua Li
admin_assigned
2025-03-14T10:56:10.748Z
658bb7e47459b6e471b9d2e6
/avatars/efd8051b468b4dbcb5d149479de67c58.svg
Wanhua Li
user
EthanTaylor
67d3adf57360ea908cf5f0bd
Renping Zhou
claimed_verified
2025-03-14T08:56:08.159Z
66f0d2036a483077eed42bfb
/avatars/f7f3f726842c26b8e52c9bdd48774b8e.svg
Renping Zhou
user
rpzhou
67d3adf57360ea908cf5f0be
Jiawei Zhou
67d3adf57360ea908cf5f0bf
Yingwei Song
admin_assigned
2025-03-14T10:56:39.206Z
6578f49ba87010c9f8a30e2a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/9XvodI5B4-ghh1dJr84lQ.jpeg
Yingwei Song
user
wrencanfly
67d3adf57360ea908cf5f0c0
Johannes Herter
67d3adf57360ea908cf5f0c1
Minghan Qin
admin_assigned
2025-03-14T10:56:49.663Z
642002b51ccd411979d72b18
https://cdn-avatars.huggingface.co/v1/production/uploads/642002b51ccd411979d72b18/JO9e0o8fAFNC-eYFbx_JW.png
Minghan Qin
user
Qmh
67d3adf57360ea908cf5f0c2
Gao Huang
67d3adf57360ea908cf5f0c3
Hanspeter Pfister
admin_assigned
2025-03-14T10:56:55.960Z
62acc69e36f7c7b7f65fccca
https://cdn-avatars.huggingface.co/v1/production/uploads/62acc69e36f7c7b7f65fccca/S8o0XE6TaQwLU8q3QPkct.png
Hanspeter Pfister
user
hpfister
4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large
  Language Models
Learning 4D language fields to enable time-sensitive, open-ended language
queries in dynamic scenes is essential for many real-world applications. While
LangSplat successfully grounds CLIP features into 3D Gaussian representations,
achieving precision and efficiency in 3D static scenes, it lacks the ability to
handle dynamic 4D fields as CLIP, designed for static image-text tasks, cannot
capture temporal dynamics in videos. Real-world environments are inherently
dynamic, with object semantics evolving over time. Building a precise 4D
language field necessitates obtaining pixel-aligned, object-wise video
features, which current vision models struggle to achieve. To address these
challenges, we propose 4D LangSplat, which learns 4D language fields to handle
time-agnostic or time-sensitive open-vocabulary queries in dynamic scenes
efficiently. 4D LangSplat bypasses learning the language field from vision
features and instead learns directly from text generated from object-wise video
captions via Multimodal Large Language Models (MLLMs). Specifically, we propose
a multimodal object-wise video prompting method, consisting of visual and text
prompts that guide MLLMs to generate detailed, temporally consistent,
high-quality captions for objects throughout a video. These captions are
encoded using a Large Language Model into high-quality sentence embeddings,
which then serve as pixel-aligned, object-specific feature supervision,
facilitating open-vocabulary text queries through shared embedding spaces.
Recognizing that objects in 4D scenes exhibit smooth transitions across states,
we further propose a status deformable network to model these continuous
changes over time effectively. Our results across multiple benchmarks
demonstrate that 4D LangSplat attains precise and efficient results for both
time-sensitive and time-agnostic open-vocabulary queries.
A new method, 4D LangSplat, uses multimodal object-wise video captions and a status deformable network to generate precise and efficient 4D language fields for dynamic scenes.
LangSplat CLIP features 3D Gaussian representations 4D language fields dynamic 4D fields MLLMs multimodal object-wise video prompting Large Language Model sentence embeddings pixel-aligned object-specific feature supervision status deformable network open-vocabulary queries