Dual-Stream Diffusion Net for Text-to-Video Generation
With the emerging diffusion models, recently, text-to-video generation has
aroused increasing attention. But an important bottleneck therein is that
generative videos often tend to carry some flickers and artifacts. In this
work, we propose a dual-stream diffusion net (DSDN) to improve the consistency
of content variations in generating videos. In particular, the designed two
diffusion streams, video content and motion branches, could not only run
separately in their private spaces for producing personalized video variations
as well as content, but also be well-aligned between the content and motion
domains through leveraging our designed cross-transformer interaction module,
which would benefit the smoothness of generated videos. Besides, we also
introduce motion decomposer and combiner to faciliate the operation on video
motion. Qualitative and quantitative experiments demonstrate that our method
could produce amazing continuous videos with fewer flickers.
64ddc0c04d6048abd76ecb9b
Binhui Liu
64ddc0c04d6048abd76ecb9c
Xin Liu
64ddc0c04d6048abd76ecb9d
Anbo Dai
admin_assigned
2023-08-17T14:08:16.806Z
628ba40d488ff0ea1c71d3cd
/avatars/b6b99150dca9006aef54516347d06166.svg
daianbo
user
daianbo
64ddc0c04d6048abd76ecb9e
Zhiyong Zeng
64ddc0c04d6048abd76ecb9f
Zhen Cui
64ddc0c04d6048abd76ecba0
Jian Yang
Dual-Stream Diffusion Net for Text-to-Video Generation
With the emerging diffusion models, recently, text-to-video generation has
aroused increasing attention. But an important bottleneck therein is that
generative videos often tend to carry some flickers and artifacts. In this
work, we propose a dual-stream diffusion net (DSDN) to improve the consistency
of content variations in generating videos. In particular, the designed two
diffusion streams, video content and motion branches, could not only run
separately in their private spaces for producing personalized video variations
as well as content, but also be well-aligned between the content and motion
domains through leveraging our designed cross-transformer interaction module,
which would benefit the smoothness of generated videos. Besides, we also
introduce motion decomposer and combiner to faciliate the operation on video
motion. Qualitative and quantitative experiments demonstrate that our method
could produce amazing continuous videos with fewer flickers.
The dual-stream diffusion net (DSDN) enhances video consistency and smoothness in text-to-video generation by using separate content and motion diffusion streams with a cross-transformer interaction module and motion decomposer/combiner.
dual-stream diffusion net (DSDN) diffusion streams video content motion branches cross-transformer interaction module motion decomposer motion combiner