Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video
  Generation Control
Diffusion models have demonstrated impressive performance in generating
high-quality videos from text prompts or images. However, precise control over
the video generation process, such as camera manipulation or content editing,
remains a significant challenge. Existing methods for controlled video
generation are typically limited to a single control type, lacking the
flexibility to handle diverse control demands. In this paper, we introduce
Diffusion as Shader (DaS), a novel approach that supports multiple video
control tasks within a unified architecture. Our key insight is that achieving
versatile video control necessitates leveraging 3D control signals, as videos
are fundamentally 2D renderings of dynamic 3D content. Unlike prior methods
limited to 2D control signals, DaS leverages 3D tracking videos as control
inputs, making the video diffusion process inherently 3D-aware. This innovation
allows DaS to achieve a wide range of video controls by simply manipulating the
3D tracking videos. A further advantage of using 3D tracking videos is their
ability to effectively link frames, significantly enhancing the temporal
consistency of the generated videos. With just 3 days of fine-tuning on 8 H800
GPUs using less than 10k videos, DaS demonstrates strong control capabilities
across diverse tasks, including mesh-to-video generation, camera control,
motion transfer, and object manipulation.
677e11b751b4ae6d3c98fdca
Zekai Gu
677e11b751b4ae6d3c98fdcb
Rui Yan
admin_assigned
2025-01-08T13:47:05.657Z
64be3a355b8d826146ed2001
/avatars/85a87270e7c76345e555803ab31c33d1.svg
ruiyan
user
ruiyan
677e11b751b4ae6d3c98fdcc
Jiahao Lu
admin_assigned
2025-01-08T13:46:55.864Z
64d3abeb1058ae59738ba8ce
/avatars/4a9de3db835ade2c20f0d2de678e85c4.svg
jiahao lu
user
jiahao97
677e11b751b4ae6d3c98fdcd
Peng Li
claimed_verified
2025-01-08T09:44:29.526Z
634e6bdd2cd84978c48f3985
/avatars/2f6ae1f50666ca187a7a3c74b5e173f7.svg
Peng Li
user
pengHTYX
677e11b751b4ae6d3c98fdce
Zhiyang Dou
admin_assigned
2025-01-08T13:47:13.361Z
645223fb01d7bd9555ea399a
https://cdn-avatars.huggingface.co/v1/production/uploads/645223fb01d7bd9555ea399a/fVR7XmGg6pMSRKx9sEdvT.png
Zhiyang Dou
user
frankzydou
677e11b751b4ae6d3c98fdcf
Chenyang Si
admin_assigned
2025-01-08T13:47:21.815Z
635f8ed47c05eb9f59963d3a
https://cdn-avatars.huggingface.co/v1/production/uploads/635f8ed47c05eb9f59963d3a/uQf4p9N9pSaFy87Wg9v4k.jpeg
ChenyangSi
user
ChenyangSi
677e11b751b4ae6d3c98fdd0
Zhen Dong
admin_assigned
2025-01-08T13:47:32.762Z
643ba2f725681c3afaa8f05e
https://cdn-avatars.huggingface.co/v1/production/uploads/643ba2f725681c3afaa8f05e/2RnOdmbBM8WHYhiTGG-Cd.jpeg
Zhen Dong
user
zhendongucb
677e11b751b4ae6d3c98fdd1
Qifeng Liu
677e11b751b4ae6d3c98fdd2
Cheng Lin
677e11b751b4ae6d3c98fdd3
Ziwei Liu
admin_assigned
2025-01-08T13:48:08.642Z
62ab1ac1d48b4d8b048a3473
https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
Ziwei Liu
user
liuziwei7
677e11b751b4ae6d3c98fdd4
Wenping Wang
677e11b751b4ae6d3c98fdd5
Yuan Liu
Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video
  Generation Control
Diffusion models have demonstrated impressive performance in generating
high-quality videos from text prompts or images. However, precise control over
the video generation process, such as camera manipulation or content editing,
remains a significant challenge. Existing methods for controlled video
generation are typically limited to a single control type, lacking the
flexibility to handle diverse control demands. In this paper, we introduce
Diffusion as Shader (DaS), a novel approach that supports multiple video
control tasks within a unified architecture. Our key insight is that achieving
versatile video control necessitates leveraging 3D control signals, as videos
are fundamentally 2D renderings of dynamic 3D content. Unlike prior methods
limited to 2D control signals, DaS leverages 3D tracking videos as control
inputs, making the video diffusion process inherently 3D-aware. This innovation
allows DaS to achieve a wide range of video controls by simply manipulating the
3D tracking videos. A further advantage of using 3D tracking videos is their
ability to effectively link frames, significantly enhancing the temporal
consistency of the generated videos. With just 3 days of fine-tuning on 8 H800
GPUs using less than 10k videos, DaS demonstrates strong control capabilities
across diverse tasks, including mesh-to-video generation, camera control,
motion transfer, and object manipulation.
Diffusion as Shader (DaS) achieves versatile video control by leveraging 3D tracking signals, enabling tasks like mesh-to-video generation, camera control, motion transfer, and object manipulation.
diffusion models video generation 3D control signals 3D tracking videos mesh-to-video generation camera control motion transfer object manipulation