PEEKABOO: Interactive Video Generation via Masked-Diffusion
Recently there has been a lot of progress in text-to-video generation, with
state-of-the-art models being capable of generating high quality, realistic
videos. However, these models lack the capability for users to interactively
control and generate videos, which can potentially unlock new areas of
application. As a first step towards this goal, we tackle the problem of
endowing diffusion-based video generation models with interactive
spatio-temporal control over their output. To this end, we take inspiration
from the recent advances in segmentation literature to propose a novel
spatio-temporal masked attention module - Peekaboo. This module is a
training-free, no-inference-overhead addition to off-the-shelf video generation
models which enables spatio-temporal control. We also propose an evaluation
benchmark for the interactive video generation task. Through extensive
qualitative and quantitative evaluation, we establish that Peekaboo enables
control video generation and even obtains a gain of upto 3.8x in mIoU over
baseline models.
6579225f5a1c60302fd1ea0f
Yash Jain
claimed_verified
2023-12-14T10:34:46.985Z
629c38f8ba19b79d354d009a
/avatars/2b37db99a4e3a2d6b86ddcbde7e2f97b.svg
Yash Jain
user
jingalala
6579225f5a1c60302fd1ea10
Anshul Nasery
claimed_verified
2023-12-13T08:32:49.032Z
65762f34ca03b6c514068e1e
/avatars/2cd0a84e67ba9bad9fa969e7bd3ca886.svg
Anshul Nasery
user
anshuln
6579225f5a1c60302fd1ea11
Vibhav Vineet
admin_assigned
2023-12-13T10:55:36.620Z
63c8527becdb7c9fdd9cacc6
/avatars/c8a3f5e1e5159ae5ead41bd9fc2b9b34.svg
Vibhav Vineet
user
vibhav-vineet
6579225f5a1c60302fd1ea12
Harkirat Behl
claimed_verified
2023-12-14T10:34:49.711Z
63c6aa9e656e7822e2359d9c
https://cdn-avatars.huggingface.co/v1/production/uploads/63c6aa9e656e7822e2359d9c/4rn_faJ23m_a6bevXEDG_.jpeg
Harkirat Behl
user
Harkirat
PEEKABOO: Interactive Video Generation via Masked-Diffusion
Recently there has been a lot of progress in text-to-video generation, with
state-of-the-art models being capable of generating high quality, realistic
videos. However, these models lack the capability for users to interactively
control and generate videos, which can potentially unlock new areas of
application. As a first step towards this goal, we tackle the problem of
endowing diffusion-based video generation models with interactive
spatio-temporal control over their output. To this end, we take inspiration
from the recent advances in segmentation literature to propose a novel
spatio-temporal masked attention module - Peekaboo. This module is a
training-free, no-inference-overhead addition to off-the-shelf video generation
models which enables spatio-temporal control. We also propose an evaluation
benchmark for the interactive video generation task. Through extensive
qualitative and quantitative evaluation, we establish that Peekaboo enables
control video generation and even obtains a gain of upto 3.8x in mIoU over
baseline models.
Peekaboo, a novel spatio-temporal masked attention module, enhances diffusion-based video generation models with interactive control, outperforming baselines in quality.
diffusion-based video generation models spatio-temporal masked attention module Peekaboo mIoU