Generative Image Dynamics
We present an approach to modeling an image-space prior on scene dynamics.
Our prior is learned from a collection of motion trajectories extracted from
real video sequences containing natural, oscillating motion such as trees,
flowers, candles, and clothes blowing in the wind. Given a single image, our
trained model uses a frequency-coordinated diffusion sampling process to
predict a per-pixel long-term motion representation in the Fourier domain,
which we call a neural stochastic motion texture. This representation can be
converted into dense motion trajectories that span an entire video. Along with
an image-based rendering module, these trajectories can be used for a number of
downstream applications, such as turning still images into seamlessly looping
dynamic videos, or allowing users to realistically interact with objects in
real pictures.
6503f7e5dcfe8fd06a78a6a2
Zhengqi Li
admin_assigned
2023-09-15T09:15:06.802Z
6384f232cafe4e4ac406d116
/avatars/33453dad4609231ba956811798e0b357.svg
Zhengqi Li
user
lixx2938
6503f7e5dcfe8fd06a78a6a3
Richard Tucker
admin_assigned
2023-09-15T09:15:14.466Z
63894f08d68e37abd59fed5c
/avatars/ea3ac13ab6289005b5b082aa88c00238.svg
Richard Tucker
user
reyet
6503f7e5dcfe8fd06a78a6a4
Noah Snavely
admin_assigned
2023-09-15T09:15:21.449Z
632b79ca0a9dc02985af9e0b
/avatars/e2ca585398a3212b2956a14804d0ba67.svg
Noah Snavely
user
jimantha
6503f7e5dcfe8fd06a78a6a5
Aleksander Holynski
admin_assigned
2023-09-15T09:15:28.222Z
62f6e2792e53c2efd33faa92
/avatars/85d94b4022577747b8d2d10a82c2f3c7.svg
Aleksander Holynski
user
holynski
Generative Image Dynamics
We present an approach to modeling an image-space prior on scene dynamics.
Our prior is learned from a collection of motion trajectories extracted from
real video sequences containing natural, oscillating motion such as trees,
flowers, candles, and clothes blowing in the wind. Given a single image, our
trained model uses a frequency-coordinated diffusion sampling process to
predict a per-pixel long-term motion representation in the Fourier domain,
which we call a neural stochastic motion texture. This representation can be
converted into dense motion trajectories that span an entire video. Along with
an image-based rendering module, these trajectories can be used for a number of
downstream applications, such as turning still images into seamlessly looping
dynamic videos, or allowing users to realistically interact with objects in
real pictures.
A frequency-coordinated diffusion sampling process is used to predict long-term motion representations for still images, enabling dynamic video creation and interactive scene manipulation.
frequency-coordinated diffusion sampling process neural stochastic motion texture Fourier domain dense motion trajectories image-based rendering module dynamic videos interactive scene manipulation