DropletVideo: A Dataset and Approach to Explore Integral Spatio-Temporal
  Consistent Video Generation
Spatio-temporal consistency is a critical research topic in video generation.
A qualified generated video segment must ensure plot plausibility and coherence
while maintaining visual consistency of objects and scenes across varying
viewpoints. Prior research, especially in open-source projects, primarily
focuses on either temporal or spatial consistency, or their basic combination,
such as appending a description of a camera movement after a prompt without
constraining the outcomes of this movement. However, camera movement may
introduce new objects to the scene or eliminate existing ones, thereby
overlaying and affecting the preceding narrative. Especially in videos with
numerous camera movements, the interplay between multiple plots becomes
increasingly complex. This paper introduces and examines integral
spatio-temporal consistency, considering the synergy between plot progression
and camera techniques, and the long-term impact of prior content on subsequent
generation. Our research encompasses dataset construction through to the
development of the model. Initially, we constructed a DropletVideo-10M dataset,
which comprises 10 million videos featuring dynamic camera motion and object
actions. Each video is annotated with an average caption of 206 words,
detailing various camera movements and plot developments. Following this, we
developed and trained the DropletVideo model, which excels in preserving
spatio-temporal coherence during video generation. The DropletVideo dataset and
model are accessible at https://dropletx.github.io.
67cfd2d7bc539099da9ebecb
Runze Zhang
67cfd2d7bc539099da9ebecc
Guoguang Du
claimed_verified
2025-03-17T08:45:36.478Z
6474a63f7d131daf633d10f2
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/rEztq682PnSt0z-UKvvad.png
GeorgeDu
user
georgedu
67cfd2d7bc539099da9ebecd
Xiaochuan Li
claimed_verified
2025-03-17T08:45:39.724Z
66b01dc4e48856bb718f2ba8
https://cdn-avatars.huggingface.co/v1/production/uploads/66b01dc4e48856bb718f2ba8/MHZyIDd3BEtsnH9M2BvzM.jpeg
Xiaochuan Li
user
lixiaochuan
67cfd2d7bc539099da9ebece
Qi Jia
67cfd2d7bc539099da9ebecf
Liang Jin
67cfd2d7bc539099da9ebed0
Lu Liu
claimed_verified
2025-03-17T08:45:32.476Z
66f67725cdcb9a4eaef04027
/avatars/fb5f4b467cc4d73e129fa9aa60ef344d.svg
Ellen Liu
user
EllenAP
67cfd2d7bc539099da9ebed1
Jingjing Wang
67cfd2d7bc539099da9ebed2
Cong Xu
claimed_verified
2025-03-18T09:16:48.232Z
6297889a64501abb8d002c6b
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/-tfSq05d4nkLkU_E-N75e.png
Cong Xu
user
NeilXu
67cfd2d7bc539099da9ebed3
Zhenhua Guo
67cfd2d7bc539099da9ebed4
Yaqian Zhao
67cfd2d7bc539099da9ebed5
Xiaoli Gong
67cfd2d7bc539099da9ebed6
Rengang Li
67cfd2d7bc539099da9ebed7
Baoyu Fan
DropletVideo: A Dataset and Approach to Explore Integral Spatio-Temporal
  Consistent Video Generation
Spatio-temporal consistency is a critical research topic in video generation.
A qualified generated video segment must ensure plot plausibility and coherence
while maintaining visual consistency of objects and scenes across varying
viewpoints. Prior research, especially in open-source projects, primarily
focuses on either temporal or spatial consistency, or their basic combination,
such as appending a description of a camera movement after a prompt without
constraining the outcomes of this movement. However, camera movement may
introduce new objects to the scene or eliminate existing ones, thereby
overlaying and affecting the preceding narrative. Especially in videos with
numerous camera movements, the interplay between multiple plots becomes
increasingly complex. This paper introduces and examines integral
spatio-temporal consistency, considering the synergy between plot progression
and camera techniques, and the long-term impact of prior content on subsequent
generation. Our research encompasses dataset construction through to the
development of the model. Initially, we constructed a DropletVideo-10M dataset,
which comprises 10 million videos featuring dynamic camera motion and object
actions. Each video is annotated with an average caption of 206 words,
detailing various camera movements and plot developments. Following this, we
developed and trained the DropletVideo model, which excels in preserving
spatio-temporal coherence during video generation. The DropletVideo dataset and
model are accessible at https://dropletx.github.io.
The DropletVideo model preserves spatio-temporal coherence in generated videos by considering both camera movements and plot developments, leveraging a large annotated dataset.
spatio-temporal consistency video generation plot plausibility visual consistency camera movement object actions DropletVideo-10M DropletVideo model