Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understanding
Haoran Zhou, Gim Hee Lee
Recent advancements in foundation models for 2D vision have enabled more effective analysis of dynamic scenes from monocular videos. However, despite their strong generalization capabilities, these models often lack 3D consistency, leading to spatial misalignment and temporal flickering that limit their effectiveness in dynamic 3D environments. In this paper, we present Motion4D, a novel framework that addresses these challenges by integrating 2D priors from foundation models into a unified 4D scene representation. Our method features a two-part iterative optimization framework: 1) Sequential optimization which updates motion and semantic fields in consecutive stages to maintain local consistency, and 2) Global optimization which jointly refines all attributes for long-term coherence. To enhance motion accuracy, we introduce a 3D confidence map that dynamically adjusts the supervision of motion priors, and an adaptive resampling process that inserts new Gaussians into under-represented regions based on per-pixel RGB and semantic errors. Furthermore, we enhance semantic coherence through an iterative refinement process that corrects 2D semantic inconsistencies by iteratively optimizing the semantic fields and updating prompts of SAM2. Extensive evaluations demonstrate that our Motion4D significantly outperforms both 2D foundation models and existing 3D representation methods in dynamic scene understanding tasks including video object segmentation and point-based tracking. Our code will be open-sourced on acceptance.