Feature4X: Bridging Any Monocular Video to 4D Agentic AI with Versatile
  Gaussian Feature Fields
Recent advancements in 2D and multimodal models have achieved remarkable
success by leveraging large-scale training on extensive datasets. However,
extending these achievements to enable free-form interactions and high-level
semantic operations with complex 3D/4D scenes remains challenging. This
difficulty stems from the limited availability of large-scale, annotated 3D/4D
or multi-view datasets, which are crucial for generalizable vision and language
tasks such as open-vocabulary and prompt-based segmentation, language-guided
editing, and visual question answering (VQA). In this paper, we introduce
Feature4X, a universal framework designed to extend any functionality from 2D
vision foundation model into the 4D realm, using only monocular video input,
which is widely available from user-generated content. The "X" in Feature4X
represents its versatility, enabling any task through adaptable,
model-conditioned 4D feature field distillation. At the core of our framework
is a dynamic optimization strategy that unifies multiple model capabilities
into a single representation. Additionally, to the best of our knowledge,
Feature4X is the first method to distill and lift the features of video
foundation models (e.g. SAM2, InternVideo2) into an explicit 4D feature field
using Gaussian Splatting. Our experiments showcase novel view segment anything,
geometric and appearance scene editing, and free-form VQA across all time
steps, empowered by LLMs in feedback loops. These advancements broaden the
scope of agentic AI applications by providing a foundation for scalable,
contextually and spatiotemporally aware systems capable of immersive dynamic 4D
scene interaction.
67e6019af54a34f8a989d8eb
Shijie Zhou
claimed_verified
2025-03-28T08:37:59.022Z
642a276516d4d8293c9a47e8
/avatars/80e6db8bc2544f3486b11b57858a8692.svg
Shijie Zhou
user
shijiezhou
67e6019af54a34f8a989d8ec
Hui Ren
claimed_verified
2025-04-03T08:29:12.621Z
65c30a5ffc592f2e565ea69f
/avatars/5b7b42fb96cc88dbeca530fedf3c2571.svg
Hui Ren
user
rhfeiyang
67e6019af54a34f8a989d8ed
Yijia Weng
67e6019af54a34f8a989d8ee
Shuwang Zhang
admin_assigned
2025-03-28T09:13:30.134Z
67b3c26f3d0f54ab3805954d
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/beyNv649ZZ5A29CH3m8mj.png
Shuwang Zhang
user
ShuwangZhang00
67e6019af54a34f8a989d8ef
Zhen Wang
67e6019af54a34f8a989d8f0
Dejia Xu
admin_assigned
2025-03-28T09:13:40.451Z
62f687a0c58915315c4ff75d
/avatars/b657180c7666735062782edd4f6a69c9.svg
Dejia Xu
user
ir1d
67e6019af54a34f8a989d8f1
Zhiwen Fan
admin_assigned
2025-03-28T09:16:41.739Z
6526386e1c6a09292d8d0a22
/avatars/471de830de2d775d35368678c1579f87.svg
fan
user
Fanzhiwen
67e6019af54a34f8a989d8f2
Suya You
67e6019af54a34f8a989d8f3
Zhangyang Wang
67e6019af54a34f8a989d8f4
Leonidas Guibas
67e6019af54a34f8a989d8f5
Achuta Kadambi
Feature4X: Bridging Any Monocular Video to 4D Agentic AI with Versatile
  Gaussian Feature Fields
Recent advancements in 2D and multimodal models have achieved remarkable
success by leveraging large-scale training on extensive datasets. However,
extending these achievements to enable free-form interactions and high-level
semantic operations with complex 3D/4D scenes remains challenging. This
difficulty stems from the limited availability of large-scale, annotated 3D/4D
or multi-view datasets, which are crucial for generalizable vision and language
tasks such as open-vocabulary and prompt-based segmentation, language-guided
editing, and visual question answering (VQA). In this paper, we introduce
Feature4X, a universal framework designed to extend any functionality from 2D
vision foundation model into the 4D realm, using only monocular video input,
which is widely available from user-generated content. The "X" in Feature4X
represents its versatility, enabling any task through adaptable,
model-conditioned 4D feature field distillation. At the core of our framework
is a dynamic optimization strategy that unifies multiple model capabilities
into a single representation. Additionally, to the best of our knowledge,
Feature4X is the first method to distill and lift the features of video
foundation models (e.g. SAM2, InternVideo2) into an explicit 4D feature field
using Gaussian Splatting. Our experiments showcase novel view segment anything,
geometric and appearance scene editing, and free-form VQA across all time
steps, empowered by LLMs in feedback loops. These advancements broaden the
scope of agentic AI applications by providing a foundation for scalable,
contextually and spatiotemporally aware systems capable of immersive dynamic 4D
scene interaction.
Feature4X extends 2D vision foundation models to 4D scene understanding using monocular video, enabling tasks like segmentation, editing, and VQA through Gaussian Splatting and dynamic optimization.
Feature4X dynamic optimization strategy 4D feature field Gaussian Splatting SAM2 InternVideo2 novel view segmentation geometric and appearance scene editing free-form VQA agentic AI scalable contextually aware spatiotemporally aware immersive dynamic 4D scene interaction