4K4DGen: Panoramic 4D Generation at 4K Resolution
The blooming of virtual reality and augmented reality (VR/AR) technologies
has driven an increasing demand for the creation of high-quality, immersive,
and dynamic environments. However, existing generative techniques either focus
solely on dynamic objects or perform outpainting from a single perspective
image, failing to meet the needs of VR/AR applications. In this work, we tackle
the challenging task of elevating a single panorama to an immersive 4D
experience. For the first time, we demonstrate the capability to generate
omnidirectional dynamic scenes with 360-degree views at 4K resolution, thereby
providing an immersive user experience. Our method introduces a pipeline that
facilitates natural scene animations and optimizes a set of 4D Gaussians using
efficient splatting techniques for real-time exploration. To overcome the lack
of scene-scale annotated 4D data and models, especially in panoramic formats,
we propose a novel Panoramic Denoiser that adapts generic 2D diffusion priors
to animate consistently in 360-degree images, transforming them into panoramic
videos with dynamic scenes at targeted regions. Subsequently, we elevate the
panoramic video into a 4D immersive environment while preserving spatial and
temporal consistency. By transferring prior knowledge from 2D models in the
perspective domain to the panoramic domain and the 4D lifting with spatial
appearance and geometry regularization, we achieve high-quality Panorama-to-4D
generation at a resolution of (4096 times 2048) for the first time. See the
project website at https://4k4dgen.github.io.
6676dd829329dbc7eaf2a41b
Renjie Li
claimed_verified
2024-06-24T07:29:57.080Z
66269d9849b603e2487c244d
/avatars/cd782c35417cd71b94c37b3216dfc3ea.svg
rj.lee
user
Rj-Lee
6676dd829329dbc7eaf2a41c
Panwang Pan
claimed_verified
2025-02-07T09:59:07.484Z
654866e8cd0a5621395f8287
https://cdn-avatars.huggingface.co/v1/production/uploads/654866e8cd0a5621395f8287/4Bccwd1ehn-Ee4T1rId5S.jpeg
Panwang Pan
user
paulpanwang
6676dd829329dbc7eaf2a41d
Bangbang Yang
claimed_verified
2025-04-22T10:17:43.838Z
63d748ff6f49aa82306b7e48
/avatars/9f0b8b8a09b14d76e52ed1bd312e6b63.svg
BB Yang
user
ybbbbt
6676dd829329dbc7eaf2a41e
Dejia Xu
claimed_verified
2024-10-15T09:11:49.057Z
62f687a0c58915315c4ff75d
/avatars/b657180c7666735062782edd4f6a69c9.svg
Dejia Xu
user
ir1d
6676dd829329dbc7eaf2a41f
Shijie Zhou
claimed_verified
2025-02-04T09:40:09.825Z
642a276516d4d8293c9a47e8
/avatars/80e6db8bc2544f3486b11b57858a8692.svg
Shijie Zhou
user
shijiezhou
6676dd829329dbc7eaf2a420
Xuanyang Zhang
6676dd829329dbc7eaf2a421
Zeming Li
6676dd829329dbc7eaf2a422
Achuta Kadambi
6676dd829329dbc7eaf2a423
Zhangyang Wang
6676dd829329dbc7eaf2a424
Zhiwen Fan
4K4DGen: Panoramic 4D Generation at 4K Resolution
The blooming of virtual reality and augmented reality (VR/AR) technologies
has driven an increasing demand for the creation of high-quality, immersive,
and dynamic environments. However, existing generative techniques either focus
solely on dynamic objects or perform outpainting from a single perspective
image, failing to meet the needs of VR/AR applications. In this work, we tackle
the challenging task of elevating a single panorama to an immersive 4D
experience. For the first time, we demonstrate the capability to generate
omnidirectional dynamic scenes with 360-degree views at 4K resolution, thereby
providing an immersive user experience. Our method introduces a pipeline that
facilitates natural scene animations and optimizes a set of 4D Gaussians using
efficient splatting techniques for real-time exploration. To overcome the lack
of scene-scale annotated 4D data and models, especially in panoramic formats,
we propose a novel Panoramic Denoiser that adapts generic 2D diffusion priors
to animate consistently in 360-degree images, transforming them into panoramic
videos with dynamic scenes at targeted regions. Subsequently, we elevate the
panoramic video into a 4D immersive environment while preserving spatial and
temporal consistency. By transferring prior knowledge from 2D models in the
perspective domain to the panoramic domain and the 4D lifting with spatial
appearance and geometry regularization, we achieve high-quality Panorama-to-4D
generation at a resolution of (4096 times 2048) for the first time. See the
project website at https://4k4dgen.github.io.
A method generates high-resolution, 360-degree, dynamic 4D panoramic videos from static panoramas, enhancing VR/AR experiences through efficient Gaussian splatting and diffusion priors.
diffusion priors natural scene animations efficient splatting techniques 4D Gaussians panoramic denoiser spatial appearance regularization geometry regularization Panorama-to-4D generation