MaskGWM: A Generalizable Driving World Model with Video Mask
  Reconstruction
World models that forecast environmental changes from actions are vital for
autonomous driving models with strong generalization. The prevailing driving
world model mainly build on video prediction model. Although these models can
produce high-fidelity video sequences with advanced diffusion-based generator,
they are constrained by their predictive duration and overall generalization
capabilities. In this paper, we explore to solve this problem by combining
generation loss with MAE-style feature-level context learning. In particular,
we instantiate this target with three key design: (1) A more scalable Diffusion
Transformer (DiT) structure trained with extra mask construction task. (2) we
devise diffusion-related mask tokens to deal with the fuzzy relations between
mask reconstruction and generative diffusion process. (3) we extend mask
construction task to spatial-temporal domain by utilizing row-wise mask for
shifted self-attention rather than masked self-attention in MAE. Then, we adopt
a row-wise cross-view module to align with this mask design. Based on above
improvement, we propose MaskGWM: a Generalizable driving World Model embodied
with Video Mask reconstruction. Our model contains two variants: MaskGWM-long,
focusing on long-horizon prediction, and MaskGWM-mview, dedicated to multi-view
generation. Comprehensive experiments on standard benchmarks validate the
effectiveness of the proposed method, which contain normal validation of
Nuscene dataset, long-horizon rollout of OpenDV-2K dataset and zero-shot
validation of Waymo dataset. Quantitative metrics on these datasets show our
method notably improving state-of-the-art driving world model.
67b705d2ebee4662205c47f7
Jingcheng Ni
admin_assigned
2025-02-24T13:06:17.841Z
65d444b1ea28ba508b87ab01
/avatars/5836c0d64ba3936e064faa8ff4d44de0.svg
Jingcheng Ni
user
kiranjc
67b705d2ebee4662205c47f8
Yuxin Guo
67b705d2ebee4662205c47f9
Yichen Liu
claimed_verified
2025-02-24T09:23:40.466Z
6572dcc6bbd6664053b1fa6b
/avatars/aba29efd00bc41f14ce422f7807cd2c3.svg
Liu Yichen
user
lyclyc52
67b705d2ebee4662205c47fa
Rui Chen
67b705d2ebee4662205c47fb
Lewei Lu
admin_assigned
2025-02-24T13:06:46.453Z
65ead3ea908526a39082e641
/avatars/dcf870695fd56b06ca03d82f831e9019.svg
Lewei Lu
user
luotto
67b705d2ebee4662205c47fc
Zehuan Wu
claimed_verified
2025-02-21T09:59:38.956Z
65717368be66cd9b65a8201c
/avatars/fe945828eec9ded4cfa3b89d48a64d90.svg
Wu Zehuan
user
wzhgba
MaskGWM: A Generalizable Driving World Model with Video Mask
  Reconstruction
World models that forecast environmental changes from actions are vital for
autonomous driving models with strong generalization. The prevailing driving
world model mainly build on video prediction model. Although these models can
produce high-fidelity video sequences with advanced diffusion-based generator,
they are constrained by their predictive duration and overall generalization
capabilities. In this paper, we explore to solve this problem by combining
generation loss with MAE-style feature-level context learning. In particular,
we instantiate this target with three key design: (1) A more scalable Diffusion
Transformer (DiT) structure trained with extra mask construction task. (2) we
devise diffusion-related mask tokens to deal with the fuzzy relations between
mask reconstruction and generative diffusion process. (3) we extend mask
construction task to spatial-temporal domain by utilizing row-wise mask for
shifted self-attention rather than masked self-attention in MAE. Then, we adopt
a row-wise cross-view module to align with this mask design. Based on above
improvement, we propose MaskGWM: a Generalizable driving World Model embodied
with Video Mask reconstruction. Our model contains two variants: MaskGWM-long,
focusing on long-horizon prediction, and MaskGWM-mview, dedicated to multi-view
generation. Comprehensive experiments on standard benchmarks validate the
effectiveness of the proposed method, which contain normal validation of
Nuscene dataset, long-horizon rollout of OpenDV-2K dataset and zero-shot
validation of Waymo dataset. Quantitative metrics on these datasets show our
method notably improving state-of-the-art driving world model.
A generalized driving world model improves long-horizon and multi-view predictions by integrating masks with diffusion-based feature learning.
diffusion-based generator MAE-style feature-level context learning Diffusion Transformer (DiT) diffusion-related mask tokens row-wise mask shifted self-attention row-wise cross-view module MaskGWM MaskGWM-long MaskGWM-mview