LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation
With the impressive progress in diffusion-based text-to-image generation,
extending such powerful generative ability to text-to-video raises enormous
attention. Existing methods either require large-scale text-video pairs and a
large number of training resources or learn motions that are precisely aligned
with template videos. It is non-trivial to balance a trade-off between the
degree of generation freedom and the resource costs for video generation. In
our study, we present a few-shot-based tuning framework, LAMP, which enables
text-to-image diffusion model Learn A specific Motion Pattern with 8~16 videos
on a single GPU. Specifically, we design a first-frame-conditioned pipeline
that uses an off-the-shelf text-to-image model for content generation so that
our tuned video diffusion model mainly focuses on motion learning. The
well-developed text-to-image techniques can provide visually pleasing and
diverse content as generation conditions, which highly improves video quality
and generation freedom. To capture the features of temporal dimension, we
expand the pretrained 2D convolution layers of the T2I model to our novel
temporal-spatial motion learning layers and modify the attention blocks to the
temporal level. Additionally, we develop an effective inference trick,
shared-noise sampling, which can improve the stability of videos with
computational costs. Our method can also be flexibly applied to other tasks,
e.g. real-world image animation and video editing. Extensive experiments
demonstrate that LAMP can effectively learn the motion pattern on limited data
and generate high-quality videos. The code and models are available at
https://rq-wu.github.io/projects/LAMP.
652f52300af36fdc8745d25e
Ruiqi Wu
claimed_verified
2023-10-19T08:10:21.057Z
6530c9a5f4c60acf0cc9259a
/avatars/a3f5452accdf62c6a0855878f0f82852.svg
Ruiqi Wu
user
wuruiqi0722
652f52300af36fdc8745d25f
Liangyu Chen
652f52300af36fdc8745d260
Tong Yang
admin_assigned
2023-10-18T09:24:50.109Z
643d901e37ca9134b5888c51
/avatars/a2f4a7959b2effed04fb193cee405da7.svg
Tony Yang
user
Tongyang
652f52300af36fdc8745d261
Chunle Guo
652f52300af36fdc8745d262
Chongyi Li
652f52300af36fdc8745d263
Xiangyu Zhang
LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation
With the impressive progress in diffusion-based text-to-image generation,
extending such powerful generative ability to text-to-video raises enormous
attention. Existing methods either require large-scale text-video pairs and a
large number of training resources or learn motions that are precisely aligned
with template videos. It is non-trivial to balance a trade-off between the
degree of generation freedom and the resource costs for video generation. In
our study, we present a few-shot-based tuning framework, LAMP, which enables
text-to-image diffusion model Learn A specific Motion Pattern with 8~16 videos
on a single GPU. Specifically, we design a first-frame-conditioned pipeline
that uses an off-the-shelf text-to-image model for content generation so that
our tuned video diffusion model mainly focuses on motion learning. The
well-developed text-to-image techniques can provide visually pleasing and
diverse content as generation conditions, which highly improves video quality
and generation freedom. To capture the features of temporal dimension, we
expand the pretrained 2D convolution layers of the T2I model to our novel
temporal-spatial motion learning layers and modify the attention blocks to the
temporal level. Additionally, we develop an effective inference trick,
shared-noise sampling, which can improve the stability of videos with
computational costs. Our method can also be flexibly applied to other tasks,
e.g. real-world image animation and video editing. Extensive experiments
demonstrate that LAMP can effectively learn the motion pattern on limited data
and generate high-quality videos. The code and models are available at
https://rq-wu.github.io/projects/LAMP.
LAMP framework enables text-to-video generation using few-shot tuning of text-to-image diffusion models with limited resources, focusing on motion learning while leveraging existing content generation techniques.
diffusion-based text-to-image text-to-video few-shot-based tuning first-frame-conditioned off-the-shelf content generation motion learning temporal-spatial motion learning layers attention blocks shared-noise sampling real-world image animation video editing