4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion
  Models
Existing dynamic scene generation methods mostly rely on distilling knowledge
from pre-trained 3D generative models, which are typically fine-tuned on
synthetic object datasets. As a result, the generated scenes are often
object-centric and lack photorealism. To address these limitations, we
introduce a novel pipeline designed for photorealistic text-to-4D scene
generation, discarding the dependency on multi-view generative models and
instead fully utilizing video generative models trained on diverse real-world
datasets. Our method begins by generating a reference video using the video
generation model. We then learn the canonical 3D representation of the video
using a freeze-time video, delicately generated from the reference video. To
handle inconsistencies in the freeze-time video, we jointly learn a per-frame
deformation to model these imperfections. We then learn the temporal
deformation based on the canonical representation to capture dynamic
interactions in the reference video. The pipeline facilitates the generation of
dynamic scenes with enhanced photorealism and structural integrity, viewable
from multiple perspectives, thereby setting a new standard in 4D scene
generation.
6669056f61588d66710d13ba
Heng Yu
6669056f61588d66710d13bb
Chaoyang Wang
admin_assigned
2024-06-12T08:17:01.607Z
64c166de11f3f1d23a99c6fd
/avatars/d4730f70a6f1cb51266862c7a8d54f77.svg
Chaoyang Wang
user
cwang9
6669056f61588d66710d13bc
Peiye Zhuang
admin_assigned
2024-06-12T08:17:07.735Z
63e473cf9db5da2dc1f4e9ff
/avatars/6973b4c959ef8bb72c40c611d384a78a.svg
peiye zhuang
user
Kelest
6669056f61588d66710d13bd
Willi Menapace
admin_assigned
2024-06-12T08:17:14.379Z
6315358a362e3e95ea538081
/avatars/3b089a25a87c2e83c6b23ccb5d2dc73e.svg
Willi Menapace
user
willi-menapace
6669056f61588d66710d13be
Aliaksandr Siarohin
admin_assigned
2024-06-12T08:17:23.361Z
64276311eb9a0ed86180715b
/avatars/76f933cd549f10e5e2db379de235d304.svg
Aliaksandr Siarohin
user
aliaksandr-siarohin
6669056f61588d66710d13bf
Junli Cao
admin_assigned
2024-06-12T08:17:29.415Z
63f54aa73aa49d8cb97b84bc
/avatars/c73c5870039611ab9162daad46a1ba20.svg
junli cao
user
jlcao2
6669056f61588d66710d13c0
Laszlo A Jeni
admin_assigned
2024-06-12T08:17:39.866Z
64bbd2f865b648b2df859bc6
/avatars/698d98d8b42c478a3d7584b93c6dbcc2.svg
Laszlo Jeni
user
laszlo-ai
6669056f61588d66710d13c1
Sergey Tulyakov
6669056f61588d66710d13c2
Hsin-Ying Lee
admin_assigned
2024-06-12T08:17:54.621Z
631bbd918de8e645af71eed9
/avatars/1cb94deadc288d07c571c9289085548c.svg
Hsin-Ying Lee
user
james371507
4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion
  Models
Existing dynamic scene generation methods mostly rely on distilling knowledge
from pre-trained 3D generative models, which are typically fine-tuned on
synthetic object datasets. As a result, the generated scenes are often
object-centric and lack photorealism. To address these limitations, we
introduce a novel pipeline designed for photorealistic text-to-4D scene
generation, discarding the dependency on multi-view generative models and
instead fully utilizing video generative models trained on diverse real-world
datasets. Our method begins by generating a reference video using the video
generation model. We then learn the canonical 3D representation of the video
using a freeze-time video, delicately generated from the reference video. To
handle inconsistencies in the freeze-time video, we jointly learn a per-frame
deformation to model these imperfections. We then learn the temporal
deformation based on the canonical representation to capture dynamic
interactions in the reference video. The pipeline facilitates the generation of
dynamic scenes with enhanced photorealism and structural integrity, viewable
from multiple perspectives, thereby setting a new standard in 4D scene
generation.
A novel pipeline for photorealistic text-to-4D scene generation uses video generative models and learns canonical 3D and temporal deformations to enhance photorealism and structural integrity.
3D generative models video generative models freeze-time video per-frame deformation temporal deformation 4D scene generation