TesserAct: Learning 4D Embodied World Models
This paper presents an effective approach for learning novel 4D embodied
world models, which predict the dynamic evolution of 3D scenes over time in
response to an embodied agent's actions, providing both spatial and temporal
consistency. We propose to learn a 4D world model by training on RGB-DN (RGB,
Depth, and Normal) videos. This not only surpasses traditional 2D models by
incorporating detailed shape, configuration, and temporal changes into their
predictions, but also allows us to effectively learn accurate inverse dynamic
models for an embodied agent. Specifically, we first extend existing robotic
manipulation video datasets with depth and normal information leveraging
off-the-shelf models. Next, we fine-tune a video generation model on this
annotated dataset, which jointly predicts RGB-DN (RGB, Depth, and Normal) for
each frame. We then present an algorithm to directly convert generated RGB,
Depth, and Normal videos into a high-quality 4D scene of the world. Our method
ensures temporal and spatial coherence in 4D scene predictions from embodied
scenarios, enables novel view synthesis for embodied environments, and
facilitates policy learning that significantly outperforms those derived from
prior video-based world models.
68118c049c2765c9323de70b
Haoyu Zhen
admin_assigned
2025-04-30T10:04:19.238Z
6437c7dae282b4a48eaf065e
https://cdn-avatars.huggingface.co/v1/production/uploads/6437c7dae282b4a48eaf065e/AxodKQXyrviTFQRyjnL01.jpeg
Haoyu Zhen
user
anyeZHY
68118c049c2765c9323de70c
Qiao Sun
68118c049c2765c9323de70d
Hongxin Zhang
claimed_verified
2025-05-02T06:36:47.235Z
6319ef09bc8f3b313f7d04d1
https://cdn-avatars.huggingface.co/v1/production/uploads/1663253467220-6319ef09bc8f3b313f7d04d1.jpeg
Hongxin Zhang
user
Icefox
68118c049c2765c9323de70e
Junyan Li
68118c049c2765c9323de70f
Siyuan Zhou
68118c049c2765c9323de710
Yilun Du
admin_assigned
2025-04-30T10:03:42.041Z
63c9bd445fdc575773c732fe
/avatars/def472d1ab3fbf751225357c0932ae7e.svg
Yilun Du
user
yilundu
68118c049c2765c9323de711
Chuang Gan
TesserAct: Learning 4D Embodied World Models
This paper presents an effective approach for learning novel 4D embodied
world models, which predict the dynamic evolution of 3D scenes over time in
response to an embodied agent's actions, providing both spatial and temporal
consistency. We propose to learn a 4D world model by training on RGB-DN (RGB,
Depth, and Normal) videos. This not only surpasses traditional 2D models by
incorporating detailed shape, configuration, and temporal changes into their
predictions, but also allows us to effectively learn accurate inverse dynamic
models for an embodied agent. Specifically, we first extend existing robotic
manipulation video datasets with depth and normal information leveraging
off-the-shelf models. Next, we fine-tune a video generation model on this
annotated dataset, which jointly predicts RGB-DN (RGB, Depth, and Normal) for
each frame. We then present an algorithm to directly convert generated RGB,
Depth, and Normal videos into a high-quality 4D scene of the world. Our method
ensures temporal and spatial coherence in 4D scene predictions from embodied
scenarios, enables novel view synthesis for embodied environments, and
facilitates policy learning that significantly outperforms those derived from
prior video-based world models.
The paper introduces a 4D world model trained on RGB-DN videos to predict spatial and temporal dynamics in embodied environments, improving inverse dynamics learning, view synthesis, and policy performance.
4D world model RGB-DN videos video generation model high-quality 4D scene temporal coherence spatial coherence embodied scenarios novel view synthesis policy learning