DimensionX: Create Any 3D and 4D Scenes from a Single Image with
  Controllable Video Diffusion
In this paper, we introduce DimensionX, a framework designed to
generate photorealistic 3D and 4D scenes from just a single image with video
diffusion. Our approach begins with the insight that both the spatial structure
of a 3D scene and the temporal evolution of a 4D scene can be effectively
represented through sequences of video frames. While recent video diffusion
models have shown remarkable success in producing vivid visuals, they face
limitations in directly recovering 3D/4D scenes due to limited spatial and
temporal controllability during generation. To overcome this, we propose
ST-Director, which decouples spatial and temporal factors in video diffusion by
learning dimension-aware LoRAs from dimension-variant data. This controllable
video diffusion approach enables precise manipulation of spatial structure and
temporal dynamics, allowing us to reconstruct both 3D and 4D representations
from sequential frames with the combination of spatial and temporal dimensions.
Additionally, to bridge the gap between generated videos and real-world scenes,
we introduce a trajectory-aware mechanism for 3D generation and an
identity-preserving denoising strategy for 4D generation. Extensive experiments
on various real-world and synthetic datasets demonstrate that DimensionX
achieves superior results in controllable video generation, as well as in 3D
and 4D scene generation, compared with previous methods.
672d8febb3f7da8ed807dc6b
Wenqiang Sun
claimed_verified
2024-11-15T09:29:57.027Z
64897b1f0ec897cfe579a399
https://cdn-avatars.huggingface.co/v1/production/uploads/64897b1f0ec897cfe579a399/ICR_75b877BaSE94gjBuj.jpeg
wenq
user
wenqsun
672d8febb3f7da8ed807dc6c
Shuo Chen
claimed_verified
2024-11-15T09:29:58.991Z
6556031ab7bad186e86e96e7
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/_SjKb8h1N8URnQchqu327.jpeg
ShuoChen
user
ShuoChen20
672d8febb3f7da8ed807dc6d
Fangfu Liu
672d8febb3f7da8ed807dc6e
Zilong Chen
672d8febb3f7da8ed807dc6f
Yueqi Duan
672d8febb3f7da8ed807dc70
Jun Zhang
672d8febb3f7da8ed807dc71
Yikai Wang
extracted_confirmed
2024-11-15T17:23:45.274Z
64206b35e40f66bcd1e3c985
/avatars/a8f830d1c47702f642045399e59c9fe8.svg
Yikai Wang
user
yikaiw
DimensionX: Create Any 3D and 4D Scenes from a Single Image with
  Controllable Video Diffusion
In this paper, we introduce DimensionX, a framework designed to
generate photorealistic 3D and 4D scenes from just a single image with video
diffusion. Our approach begins with the insight that both the spatial structure
of a 3D scene and the temporal evolution of a 4D scene can be effectively
represented through sequences of video frames. While recent video diffusion
models have shown remarkable success in producing vivid visuals, they face
limitations in directly recovering 3D/4D scenes due to limited spatial and
temporal controllability during generation. To overcome this, we propose
ST-Director, which decouples spatial and temporal factors in video diffusion by
learning dimension-aware LoRAs from dimension-variant data. This controllable
video diffusion approach enables precise manipulation of spatial structure and
temporal dynamics, allowing us to reconstruct both 3D and 4D representations
from sequential frames with the combination of spatial and temporal dimensions.
Additionally, to bridge the gap between generated videos and real-world scenes,
we introduce a trajectory-aware mechanism for 3D generation and an
identity-preserving denoising strategy for 4D generation. Extensive experiments
on various real-world and synthetic datasets demonstrate that DimensionX
achieves superior results in controllable video generation, as well as in 3D
and 4D scene generation, compared with previous methods.
DimensionX leverages video diffusion with spatial-temporal decoupling and trajectory-aware mechanisms to generate high-quality 3D and 4D scenes from single images.
video diffusion ST-Director dimension-aware LoRAs spatial-temporal decoupling trajectory-aware mechanism identity-preserving denoising