3D and 4D World Modeling: A Survey
World modeling has become a cornerstone in AI research, enabling agents to
understand, represent, and predict the dynamic environments they inhabit. While
prior work largely emphasizes generative methods for 2D image and video data,
they overlook the rapidly growing body of work that leverages native 3D and 4D
representations such as RGB-D imagery, occupancy grids, and LiDAR point clouds
for large-scale scene modeling. At the same time, the absence of a standardized
definition and taxonomy for ``world models'' has led to fragmented and
sometimes inconsistent claims in the literature. This survey addresses these
gaps by presenting the first comprehensive review explicitly dedicated to 3D
and 4D world modeling and generation. We establish precise definitions,
introduce a structured taxonomy spanning video-based (VideoGen),
occupancy-based (OccGen), and LiDAR-based (LiDARGen) approaches, and
systematically summarize datasets and evaluation metrics tailored to 3D/4D
settings. We further discuss practical applications, identify open challenges,
and highlight promising research directions, aiming to provide a coherent and
foundational reference for advancing the field. A systematic summary of
existing literature is available at https://github.com/worldbench/survey
68c23b0629b8ec9932cd095b
Lingdong Kong
claimed_verified
2025-09-12T16:13:34.754Z
62df78222d89ce551ce0f71d
/avatars/89fba294cff2d2f941d121c1923e4c76.svg
Lingdong Kong
user
ldkong
68c23b0629b8ec9932cd095c
Wesley Yang
68c23b0629b8ec9932cd095d
Jianbiao Mei
68c23b0629b8ec9932cd095e
Youquan Liu
68c23b0629b8ec9932cd095f
Ao Liang
68c23b0629b8ec9932cd0960
Dekai Zhu
68c23b0629b8ec9932cd0961
Dongyue Lu
68c23b0629b8ec9932cd0962
Wei Yin
68c23b0629b8ec9932cd0963
Xiaotao Hu
68c23b0629b8ec9932cd0964
Mingkai Jia
claimed_verified
2025-09-12T16:13:28.288Z
66740d89f0bfd6fc2171d662
/avatars/b948d91e44811a8e5379f6fabb9b536a.svg
Mingkai Jia
user
mkjia
68c23b0629b8ec9932cd0965
Junyuan Deng
68c23b0629b8ec9932cd0966
Kaiwen Zhang
claimed_verified
2025-09-12T16:13:15.862Z
641087ddf52d7eb22e0673dd
/avatars/946f29569aeda2cbd9ef2b94b1ab1391.svg
Kevin Zhang
user
Kevin-thu
68c23b0629b8ec9932cd0967
Yang Wu
68c23b0629b8ec9932cd0968
Tianyi Yan
68c23b0629b8ec9932cd0969
Shenyuan Gao
68c23b0629b8ec9932cd096a
Song Wang
claimed_verified
2025-09-12T16:13:23.649Z
66863d26e2b71e3d09189ae9
/avatars/3c0e6f30e053f2e622ae75e1dc43edba.svg
Song Wang
user
songw-zju
68c23b0629b8ec9932cd096b
Linfeng Li
68c23b0629b8ec9932cd096c
Liang Pan
68c23b0629b8ec9932cd096d
Yong Liu
68c23b0629b8ec9932cd096e
Jianke Zhu
68c23b0629b8ec9932cd096f
Wei Tsang Ooi
68c23b0629b8ec9932cd0970
Steven C. H. Hoi
68c23b0629b8ec9932cd0971
Ziwei Liu
3D and 4D World Modeling: A Survey
World modeling has become a cornerstone in AI research, enabling agents to
understand, represent, and predict the dynamic environments they inhabit. While
prior work largely emphasizes generative methods for 2D image and video data,
they overlook the rapidly growing body of work that leverages native 3D and 4D
representations such as RGB-D imagery, occupancy grids, and LiDAR point clouds
for large-scale scene modeling. At the same time, the absence of a standardized
definition and taxonomy for ``world models'' has led to fragmented and
sometimes inconsistent claims in the literature. This survey addresses these
gaps by presenting the first comprehensive review explicitly dedicated to 3D
and 4D world modeling and generation. We establish precise definitions,
introduce a structured taxonomy spanning video-based (VideoGen),
occupancy-based (OccGen), and LiDAR-based (LiDARGen) approaches, and
systematically summarize datasets and evaluation metrics tailored to 3D/4D
settings. We further discuss practical applications, identify open challenges,
and highlight promising research directions, aiming to provide a coherent and
foundational reference for advancing the field. A systematic summary of
existing literature is available at https://github.com/worldbench/survey
This survey provides a comprehensive review of 3D and 4D world modeling and generation, establishing definitions, taxonomy, datasets, and evaluation metrics, and discussing applications and challenges.
world models 3D world modeling 4D world modeling generative methods RGB-D imagery occupancy grids LiDAR point clouds VideoGen OccGen LiDARGen datasets evaluation metrics applications open challenges research directions