HoloTime: Taming Video Diffusion Models for Panoramic 4D Scene
  Generation
The rapid advancement of diffusion models holds the promise of
revolutionizing the application of VR and AR technologies, which typically
require scene-level 4D assets for user experience. Nonetheless, existing
diffusion models predominantly concentrate on modeling static 3D scenes or
object-level dynamics, constraining their capacity to provide truly immersive
experiences. To address this issue, we propose HoloTime, a framework that
integrates video diffusion models to generate panoramic videos from a single
prompt or reference image, along with a 360-degree 4D scene reconstruction
method that seamlessly transforms the generated panoramic video into 4D assets,
enabling a fully immersive 4D experience for users. Specifically, to tame video
diffusion models for generating high-fidelity panoramic videos, we introduce
the 360World dataset, the first comprehensive collection of panoramic videos
suitable for downstream 4D scene reconstruction tasks. With this curated
dataset, we propose Panoramic Animator, a two-stage image-to-video diffusion
model that can convert panoramic images into high-quality panoramic videos.
Following this, we present Panoramic Space-Time Reconstruction, which leverages
a space-time depth estimation method to transform the generated panoramic
videos into 4D point clouds, enabling the optimization of a holistic 4D
Gaussian Splatting representation to reconstruct spatially and temporally
consistent 4D scenes. To validate the efficacy of our method, we conducted a
comparative analysis with existing approaches, revealing its superiority in
both panoramic video generation and 4D scene reconstruction. This demonstrates
our method's capability to create more engaging and realistic immersive
environments, thereby enhancing user experiences in VR and AR applications.
681b258f23dbf6b59c5fc735
Haiyang Zhou
claimed_verified
2025-05-08T08:59:30.816Z
66eeda3676a8038cb448f11d
/avatars/8d6e61f4c9354c6720ccaa7be0fe1d9f.svg
Haiyang Zhou
user
Marblueocean
681b258f23dbf6b59c5fc736
Wangbo Yu
extracted_pending
2025-05-07T09:19:13.250Z
63f095be6309c84d5f48848a
https://cdn-avatars.huggingface.co/v1/production/uploads/63f095be6309c84d5f48848a/pL2CKi-r-0mMfIGhYSAsm.jpeg
Wangbo Yu
user
Drexubery
681b258f23dbf6b59c5fc737
Jiawen Guan
681b258f23dbf6b59c5fc738
Xinhua Cheng
681b258f23dbf6b59c5fc739
Yonghong Tian
681b258f23dbf6b59c5fc73a
Li Yuan
HoloTime: Taming Video Diffusion Models for Panoramic 4D Scene
  Generation
The rapid advancement of diffusion models holds the promise of
revolutionizing the application of VR and AR technologies, which typically
require scene-level 4D assets for user experience. Nonetheless, existing
diffusion models predominantly concentrate on modeling static 3D scenes or
object-level dynamics, constraining their capacity to provide truly immersive
experiences. To address this issue, we propose HoloTime, a framework that
integrates video diffusion models to generate panoramic videos from a single
prompt or reference image, along with a 360-degree 4D scene reconstruction
method that seamlessly transforms the generated panoramic video into 4D assets,
enabling a fully immersive 4D experience for users. Specifically, to tame video
diffusion models for generating high-fidelity panoramic videos, we introduce
the 360World dataset, the first comprehensive collection of panoramic videos
suitable for downstream 4D scene reconstruction tasks. With this curated
dataset, we propose Panoramic Animator, a two-stage image-to-video diffusion
model that can convert panoramic images into high-quality panoramic videos.
Following this, we present Panoramic Space-Time Reconstruction, which leverages
a space-time depth estimation method to transform the generated panoramic
videos into 4D point clouds, enabling the optimization of a holistic 4D
Gaussian Splatting representation to reconstruct spatially and temporally
consistent 4D scenes. To validate the efficacy of our method, we conducted a
comparative analysis with existing approaches, revealing its superiority in
both panoramic video generation and 4D scene reconstruction. This demonstrates
our method's capability to create more engaging and realistic immersive
environments, thereby enhancing user experiences in VR and AR applications.
HoloTime uses a two-stage panoramic diffusion model and a space-time depth estimation method to generate high-fidelity 4D assets for immersive VR and AR experiences.
diffusion models panoramic videos 4D scene reconstruction 360World dataset Panoramic Animator two-stage image-to-video diffusion model Panoramic Space-Time Reconstruction space-time depth estimation 4D Gaussian Splatting spatially and temporally consistent 4D scenes