Searching Priors Makes Text-to-Video Synthesis Better
Significant advancements in video diffusion models have brought substantial
progress to the field of text-to-video (T2V) synthesis. However, existing T2V
synthesis model struggle to accurately generate complex motion dynamics,
leading to a reduction in video realism. One possible solution is to collect
massive data and train the model on it, but this would be extremely expensive.
To alleviate this problem, in this paper, we reformulate the typical T2V
generation process as a search-based generation pipeline. Instead of scaling up
the model training, we employ existing videos as the motion prior database.
Specifically, we divide T2V generation process into two steps: (i) For a given
prompt input, we search existing text-video datasets to find videos with text
labels that closely match the prompt motions. We propose a tailored search
algorithm that emphasizes object motion features. (ii) Retrieved videos are
processed and distilled into motion priors to fine-tune a pre-trained base T2V
model, followed by generating desired videos using input prompt. By utilizing
the priors gleaned from the searched videos, we enhance the realism of the
generated videos' motion. All operations can be finished on a single NVIDIA RTX
4090 GPU. We validate our method against state-of-the-art T2V models across
diverse prompt inputs. The code will be public.
66613546754f90a8582cfbfa
Haoran Cheng
admin_assigned
2024-06-06T10:05:26.459Z
652cd5ee66313ebb61cac09c
/avatars/fcbae1dc7134e625f2d79f767b34e142.svg
Haoran Cheng
user
chrxx98
66613546754f90a8582cfbfb
Liang Peng
claimed_verified
2024-06-06T10:53:31.404Z
66618e365fd9d736e68426d1
/avatars/76cb9044866017749c626001da1762e8.svg
PengLiang
user
PengLiang002
66613546754f90a8582cfbfc
Linxuan Xia
admin_assigned
2024-06-06T10:05:57.120Z
6459e93cb8e78d46d7e046b5
/avatars/8c944d27f934760fecd31942464ea9de.svg
Linxuan Xia
user
Charlo7te
66613546754f90a8582cfbfd
Yuepeng Hu
admin_assigned
2024-06-06T10:06:04.579Z
65ce2c9e3f1b48716ea36271
/avatars/cb238ed8549150f88207d9206787d083.svg
Yuepeng Hu
user
HifiHu
66613546754f90a8582cfbfe
Hengjia Li
claimed_verified
2024-12-04T09:47:22.910Z
64631cbd39359568c63a7fd6
/avatars/e921b6a25314bf14292ffdd0bc6f37de.svg
Li
user
Pluto5
66613546754f90a8582cfbff
Qinglin Lu
admin_assigned
2024-06-06T10:07:00.863Z
64c13276b4fd9b97756b1783
/avatars/a1d0480fa94f59ed5db3a1bba09bd660.svg
lu
user
qinglin3
66613546754f90a8582cfc00
Xiaofei He
66613546754f90a8582cfc01
Boxi Wu
Searching Priors Makes Text-to-Video Synthesis Better
Significant advancements in video diffusion models have brought substantial
progress to the field of text-to-video (T2V) synthesis. However, existing T2V
synthesis model struggle to accurately generate complex motion dynamics,
leading to a reduction in video realism. One possible solution is to collect
massive data and train the model on it, but this would be extremely expensive.
To alleviate this problem, in this paper, we reformulate the typical T2V
generation process as a search-based generation pipeline. Instead of scaling up
the model training, we employ existing videos as the motion prior database.
Specifically, we divide T2V generation process into two steps: (i) For a given
prompt input, we search existing text-video datasets to find videos with text
labels that closely match the prompt motions. We propose a tailored search
algorithm that emphasizes object motion features. (ii) Retrieved videos are
processed and distilled into motion priors to fine-tune a pre-trained base T2V
model, followed by generating desired videos using input prompt. By utilizing
the priors gleaned from the searched videos, we enhance the realism of the
generated videos' motion. All operations can be finished on a single NVIDIA RTX
4090 GPU. We validate our method against state-of-the-art T2V models across
diverse prompt inputs. The code will be public.
This method enhances text-to-video synthesis by using a search-based generation pipeline with motion priors from existing datasets, improving video realism without extensive retraining.
video diffusion models text-to-video (T2V) synthesis motion dynamics motion prior database search-based generation pipeline pre-trained base T2V model object motion features motion priors