ActAnywhere: Subject-Aware Video Background Generation
Generating video background that tailors to foreground subject motion is an
important problem for the movie industry and visual effects community. This
task involves synthesizing background that aligns with the motion and
appearance of the foreground subject, while also complies with the artist's
creative intention. We introduce ActAnywhere, a generative model that automates
this process which traditionally requires tedious manual efforts. Our model
leverages the power of large-scale video diffusion models, and is specifically
tailored for this task. ActAnywhere takes a sequence of foreground subject
segmentation as input and an image that describes the desired scene as
condition, to produce a coherent video with realistic foreground-background
interactions while adhering to the condition frame. We train our model on a
large-scale dataset of human-scene interaction videos. Extensive evaluations
demonstrate the superior performance of our model, significantly outperforming
baselines. Moreover, we show that ActAnywhere generalizes to diverse
out-of-distribution samples, including non-human subjects. Please visit our
project webpage at https://actanywhere.github.io.
65ade2dbfd4261f53104ec41
Boxiao Pan
admin_assigned
2024-01-22T13:13:11.788Z
64b5c10904fa6584c0464be2
/avatars/13398577c6689fc3340c6c4c40cb2815.svg
Boxiao Pan
user
bpan01
65ade2dbfd4261f53104ec42
Zhan Xu
65ade2dbfd4261f53104ec43
Chun-Hao Paul Huang
admin_assigned
2024-01-22T13:14:04.766Z
6318069a33a4e2b5d3b47fe5
/avatars/c9a9818e64186746557466cc7a6d57b8.svg
CHUN-HAO  HUANG
user
ccool2486
65ade2dbfd4261f53104ec44
Krishna Kumar Singh
65ade2dbfd4261f53104ec45
Yang Zhou
65ade2dbfd4261f53104ec46
Leonidas J. Guibas
65ade2dbfd4261f53104ec47
Jimei Yang
claimed_verified
2024-01-22T21:11:24.870Z
65ae93c53e82498d4cdaa428
/avatars/b03221ed402247f6e7a80efef834574f.svg
Yang
user
Jimei
ActAnywhere: Subject-Aware Video Background Generation
Generating video background that tailors to foreground subject motion is an
important problem for the movie industry and visual effects community. This
task involves synthesizing background that aligns with the motion and
appearance of the foreground subject, while also complies with the artist's
creative intention. We introduce ActAnywhere, a generative model that automates
this process which traditionally requires tedious manual efforts. Our model
leverages the power of large-scale video diffusion models, and is specifically
tailored for this task. ActAnywhere takes a sequence of foreground subject
segmentation as input and an image that describes the desired scene as
condition, to produce a coherent video with realistic foreground-background
interactions while adhering to the condition frame. We train our model on a
large-scale dataset of human-scene interaction videos. Extensive evaluations
demonstrate the superior performance of our model, significantly outperforming
baselines. Moreover, we show that ActAnywhere generalizes to diverse
out-of-distribution samples, including non-human subjects. Please visit our
project webpage at https://actanywhere.github.io.
ActAnywhere, a generative model utilizing video diffusion, automates coherent background generation for videos, aligning with foreground motion and artistic conditions.
generative model video diffusion models foreground subject segmentation condition frame foreground-background interactions human-scene interaction videos