SketchVideo: Sketch-based Video Generation and Editing
Video generation and editing conditioned on text prompts or images have
undergone significant advancements. However, challenges remain in accurately
controlling global layout and geometry details solely by texts, and supporting
motion control and local modification through images. In this paper, we aim to
achieve sketch-based spatial and motion control for video generation and
support fine-grained editing of real or synthetic videos. Based on the DiT
video generation model, we propose a memory-efficient control structure with
sketch control blocks that predict residual features of skipped DiT blocks.
Sketches are drawn on one or two keyframes (at arbitrary time points) for easy
interaction. To propagate such temporally sparse sketch conditions across all
frames, we propose an inter-frame attention mechanism to analyze the
relationship between the keyframes and each video frame. For sketch-based video
editing, we design an additional video insertion module that maintains
consistency between the newly edited content and the original video's spatial
feature and dynamic motion. During inference, we use latent fusion for the
accurate preservation of unedited regions. Extensive experiments demonstrate
that our SketchVideo achieves superior performance in controllable video
generation and editing.
67eb5280aeab4ce97de07134
Feng-Lin Liu
claimed_verified
2025-04-01T07:47:05.907Z
6424538b9f9e65b42389920e
/avatars/9b912e2af9eebe9a481181f006765059.svg
Feng-Lin Liu
user
Okrin
67eb5280aeab4ce97de07135
Hongbo Fu
admin_assigned
2025-04-01T08:07:01.616Z
662cd8b9322afcbae53fb06e
/avatars/9847f5c2282d49e61e76a0a303e0b2b1.svg
fuhongbo
user
fuhongbo
67eb5280aeab4ce97de07136
Xintao Wang
admin_assigned
2025-04-01T08:07:09.910Z
60e272ca6c78a8c122b12127
https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
Xintao Wang
user
Xintao
67eb5280aeab4ce97de07137
Weicai Ye
admin_assigned
2025-04-01T08:07:16.323Z
6360d9f0472131c3bc4f61df
/avatars/c5d884e5ef19b781e3405aba6dd68ca8.svg
WeicaiYe
user
WeicaiYe
67eb5280aeab4ce97de07138
Pengfei Wan
claimed_verified
2025-04-03T08:29:02.682Z
662f93942510ef5735d7ad00
https://cdn-avatars.huggingface.co/v1/production/uploads/662f93942510ef5735d7ad00/ZIDIPm63sncIHFTT5b0uR.png
magicwpf
user
magicwpf
67eb5280aeab4ce97de07139
Di Zhang
admin_assigned
2025-04-01T08:07:30.236Z
644c8324f02250233d0d67d9
/avatars/feb39d281457c1750f3eada3c060a23e.svg
Di Zhang
user
dizhang
67eb5280aeab4ce97de0713a
Lin Gao
SketchVideo: Sketch-based Video Generation and Editing
Video generation and editing conditioned on text prompts or images have
undergone significant advancements. However, challenges remain in accurately
controlling global layout and geometry details solely by texts, and supporting
motion control and local modification through images. In this paper, we aim to
achieve sketch-based spatial and motion control for video generation and
support fine-grained editing of real or synthetic videos. Based on the DiT
video generation model, we propose a memory-efficient control structure with
sketch control blocks that predict residual features of skipped DiT blocks.
Sketches are drawn on one or two keyframes (at arbitrary time points) for easy
interaction. To propagate such temporally sparse sketch conditions across all
frames, we propose an inter-frame attention mechanism to analyze the
relationship between the keyframes and each video frame. For sketch-based video
editing, we design an additional video insertion module that maintains
consistency between the newly edited content and the original video's spatial
feature and dynamic motion. During inference, we use latent fusion for the
accurate preservation of unedited regions. Extensive experiments demonstrate
that our SketchVideo achieves superior performance in controllable video
generation and editing.
SketchVideo enhances video generation and editing by incorporating sketch control blocks, inter-frame attention, and latent fusion, improving global layout, motion control, and consistency.
DiT memory-efficient control structure sketch control blocks keyframes inter-frame attention video insertion module latent fusion controllable video generation video editing