Matrix-3D: Omnidirectional Explorable 3D World Generation
Explorable 3D world generation from a single image or text prompt forms a
cornerstone of spatial intelligence. Recent works utilize video model to
achieve wide-scope and generalizable 3D world generation. However, existing
approaches often suffer from a limited scope in the generated scenes. In this
work, we propose Matrix-3D, a framework that utilize panoramic representation
for wide-coverage omnidirectional explorable 3D world generation that combines
conditional video generation and panoramic 3D reconstruction. We first train a
trajectory-guided panoramic video diffusion model that employs scene mesh
renders as condition, to enable high-quality and geometrically consistent scene
video generation. To lift the panorama scene video to 3D world, we propose two
separate methods: (1) a feed-forward large panorama reconstruction model for
rapid 3D scene reconstruction and (2) an optimization-based pipeline for
accurate and detailed 3D scene reconstruction. To facilitate effective
training, we also introduce the Matrix-Pano dataset, the first large-scale
synthetic collection comprising 116K high-quality static panoramic video
sequences with depth and trajectory annotations. Extensive experiments
demonstrate that our proposed framework achieves state-of-the-art performance
in panoramic video generation and 3D world generation. See more in
https://matrix-3d.github.io.
689ac0a0fab6fdd2e52ac4e0
Zhongqi Yang
claimed_verified
2025-08-14T13:39:33.067Z
6858f55c4f9d9853a2d3d12e
/avatars/a4739436deb2a0f2b327181fd31a5056.svg
Zhongqi.Yang
user
Inn1917
689ac0a0fab6fdd2e52ac4e1
Wenhang Ge
claimed_verified
2025-08-14T13:39:31.249Z
64758ee7d815855e4efa206b
/avatars/105ada08a9a982fc7b723bdc678f7e72.svg
wenhang ge
user
spongy
689ac0a0fab6fdd2e52ac4e2
Yuqi Li
689ac0a0fab6fdd2e52ac4e3
Jiaqi Chen
689ac0a0fab6fdd2e52ac4e4
Haoyuan Li
689ac0a0fab6fdd2e52ac4e5
Mengyin An
claimed_verified
2025-08-13T07:19:49.471Z
67a9664af5f1253c64259c50
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Jk9W1C8704pqxNOsqZ0d9.png
an
user
dearamy
689ac0a0fab6fdd2e52ac4e6
Fei Kang
claimed_verified
2025-08-13T07:19:46.490Z
67a9b36a2fbf63093c19d3de
https://cdn-avatars.huggingface.co/v1/production/uploads/67a9b36a2fbf63093c19d3de/LbB2XH0ezcJ_tdu4hU0Of.png
老k
user
kangfei
689ac0a0fab6fdd2e52ac4e7
Hua Xue
689ac0a0fab6fdd2e52ac4e8
Baixin Xu
689ac0a0fab6fdd2e52ac4e9
Yuyang Yin
689ac0a0fab6fdd2e52ac4ea
Eric Li
689ac0a0fab6fdd2e52ac4eb
Yang Liu
689ac0a0fab6fdd2e52ac4ec
Yikai Wang
689ac0a0fab6fdd2e52ac4ed
Hao-Xiang Guo
claimed_verified
2025-08-14T13:39:35.091Z
6658589605d3e3058e469d34
/avatars/c437e92684fa87af411a06d0c5ef6b9e.svg
haoxiang guo
user
howardguo222
689ac0a0fab6fdd2e52ac4ee
Yahui Zhou
Matrix-3D: Omnidirectional Explorable 3D World Generation
Explorable 3D world generation from a single image or text prompt forms a
cornerstone of spatial intelligence. Recent works utilize video model to
achieve wide-scope and generalizable 3D world generation. However, existing
approaches often suffer from a limited scope in the generated scenes. In this
work, we propose Matrix-3D, a framework that utilize panoramic representation
for wide-coverage omnidirectional explorable 3D world generation that combines
conditional video generation and panoramic 3D reconstruction. We first train a
trajectory-guided panoramic video diffusion model that employs scene mesh
renders as condition, to enable high-quality and geometrically consistent scene
video generation. To lift the panorama scene video to 3D world, we propose two
separate methods: (1) a feed-forward large panorama reconstruction model for
rapid 3D scene reconstruction and (2) an optimization-based pipeline for
accurate and detailed 3D scene reconstruction. To facilitate effective
training, we also introduce the Matrix-Pano dataset, the first large-scale
synthetic collection comprising 116K high-quality static panoramic video
sequences with depth and trajectory annotations. Extensive experiments
demonstrate that our proposed framework achieves state-of-the-art performance
in panoramic video generation and 3D world generation. See more in
https://matrix-3d.github.io.
Matrix-3D generates wide-coverage 3D worlds from single images or text using panoramic video diffusion and reconstruction models.
panoramic representation wide-coverage omnidirectional explorable 3D world generation conditional video generation panoramic 3D reconstruction trajectory-guided panoramic video diffusion model scene mesh renders feed-forward large panorama reconstruction model optimization-based pipeline Matrix-Pano dataset panoramic video generation 3D world generation