I4VGen: Image as Stepping Stone for Text-to-Video Generation
Text-to-video generation has lagged behind text-to-image synthesis in quality
and diversity due to the complexity of spatio-temporal modeling and limited
video-text datasets. This paper presents I4VGen, a training-free and
plug-and-play video diffusion inference framework, which enhances text-to-video
generation by leveraging robust image techniques. Specifically, following
text-to-image-to-video, I4VGen decomposes the text-to-video generation into two
stages: anchor image synthesis and anchor image-guided video synthesis.
Correspondingly, a well-designed generation-selection pipeline is employed to
achieve visually-realistic and semantically-faithful anchor image, and an
innovative Noise-Invariant Video Score Distillation Sampling is incorporated to
animate the image to a dynamic video, followed by a video regeneration process
to refine the video. This inference strategy effectively mitigates the
prevalent issue of non-zero terminal signal-to-noise ratio. Extensive
evaluations show that I4VGen not only produces videos with higher visual
realism and textual fidelity but also integrates seamlessly into existing
image-to-video diffusion models, thereby improving overall video quality.
665fe3bc22834190f7cd102a
Xiefan Guo
admin_assigned
2024-06-05T08:24:37.115Z
64905cd589f22918ecaca080
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/2S7I7uZL49CXbUN2T7p63.jpeg
Xiefan Guo
user
xiefan-guo
665fe3bc22834190f7cd102b
Jinlin Liu
665fe3bc22834190f7cd102c
Miaomiao Cui
admin_assigned
2024-06-05T08:25:44.412Z
665024770a7476925c31aab8
/avatars/ab738817124b1a5b2bd7cf9d5bacf1e9.svg
Cui_Miaomiao
user
Miaomiao0109
665fe3bc22834190f7cd102d
Di Huang
I4VGen: Image as Stepping Stone for Text-to-Video Generation
Text-to-video generation has lagged behind text-to-image synthesis in quality
and diversity due to the complexity of spatio-temporal modeling and limited
video-text datasets. This paper presents I4VGen, a training-free and
plug-and-play video diffusion inference framework, which enhances text-to-video
generation by leveraging robust image techniques. Specifically, following
text-to-image-to-video, I4VGen decomposes the text-to-video generation into two
stages: anchor image synthesis and anchor image-guided video synthesis.
Correspondingly, a well-designed generation-selection pipeline is employed to
achieve visually-realistic and semantically-faithful anchor image, and an
innovative Noise-Invariant Video Score Distillation Sampling is incorporated to
animate the image to a dynamic video, followed by a video regeneration process
to refine the video. This inference strategy effectively mitigates the
prevalent issue of non-zero terminal signal-to-noise ratio. Extensive
evaluations show that I4VGen not only produces videos with higher visual
realism and textual fidelity but also integrates seamlessly into existing
image-to-video diffusion models, thereby improving overall video quality.
I4VGen enhances text-to-video generation through image synthesis and guided video synthesis, using robust techniques to produce high-quality, realistic videos.
diffusion image synthesis anchor image anchor image-guided video synthesis generation-selection pipeline Noise-Invariant Video Score Distillation Sampling image-to-video diffusion models