Generative Rendering: Controllable 4D-Guided Video Generation with 2D
  Diffusion Models
Traditional 3D content creation tools empower users to bring their
imagination to life by giving them direct control over a scene's geometry,
appearance, motion, and camera path. Creating computer-generated videos,
however, is a tedious manual process, which can be automated by emerging
text-to-video diffusion models. Despite great promise, video diffusion models
are difficult to control, hindering a user to apply their own creativity rather
than amplifying it. To address this challenge, we present a novel approach that
combines the controllability of dynamic 3D meshes with the expressivity and
editability of emerging diffusion models. For this purpose, our approach takes
an animated, low-fidelity rendered mesh as input and injects the ground truth
correspondence information obtained from the dynamic mesh into various stages
of a pre-trained text-to-image generation model to output high-quality and
temporally consistent frames. We demonstrate our approach on various examples
where motion can be obtained by animating rigged assets or changing the camera
path.
656e8d822cd7a3e348fe00c0
Shengqu Cai
claimed_verified
2024-11-28T09:20:30.203Z
66a1da7cc9e703d2af5ad742
/avatars/f9cd9ae3407e249ab4569479200feb1f.svg
Shengqu Cai
user
primecai
656e8d822cd7a3e348fe00c1
Duygu Ceylan
656e8d822cd7a3e348fe00c2
Matheus Gadelha
admin_assigned
2023-12-05T12:09:07.528Z
6315a9c466062476ae1fa0c7
/avatars/c59749be7ecee4858c59889fea61c88a.svg
Matheus
user
matheusgadelha
656e8d822cd7a3e348fe00c3
Chun-Hao Paul Huang
admin_assigned
2023-12-05T12:09:19.593Z
6318069a33a4e2b5d3b47fe5
/avatars/c9a9818e64186746557466cc7a6d57b8.svg
CHUN-HAO  HUANG
user
ccool2486
656e8d822cd7a3e348fe00c4
Tuanfeng Yang Wang
admin_assigned
2023-12-05T12:09:42.306Z
630403daeedc089484c76b61
/avatars/30bd8d24dbf88ceb6ca823618ac47d95.svg
tuanfeng wang
user
yangtwan
656e8d822cd7a3e348fe00c5
Gordon Wetzstein
Generative Rendering: Controllable 4D-Guided Video Generation with 2D
  Diffusion Models
Traditional 3D content creation tools empower users to bring their
imagination to life by giving them direct control over a scene's geometry,
appearance, motion, and camera path. Creating computer-generated videos,
however, is a tedious manual process, which can be automated by emerging
text-to-video diffusion models. Despite great promise, video diffusion models
are difficult to control, hindering a user to apply their own creativity rather
than amplifying it. To address this challenge, we present a novel approach that
combines the controllability of dynamic 3D meshes with the expressivity and
editability of emerging diffusion models. For this purpose, our approach takes
an animated, low-fidelity rendered mesh as input and injects the ground truth
correspondence information obtained from the dynamic mesh into various stages
of a pre-trained text-to-image generation model to output high-quality and
temporally consistent frames. We demonstrate our approach on various examples
where motion can be obtained by animating rigged assets or changing the camera
path.
dynamic 3D meshes text-to-video diffusion models bit truth correspondence information pre-trained text-to-image generation model temporally consistent frames rigged assets