Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video
  Generation
We consider the task of Image-to-Video (I2V) generation, which involves
transforming static images into realistic video sequences based on a textual
description. While recent advancements produce photorealistic outputs, they
frequently struggle to create videos with accurate and consistent object
motion, especially in multi-object scenarios. To address these limitations, we
propose a two-stage compositional framework that decomposes I2V generation
into: (i) An explicit intermediate representation generation stage, followed by
(ii) A video generation stage that is conditioned on this representation. Our
key innovation is the introduction of a mask-based motion trajectory as an
intermediate representation, that captures both semantic object information and
motion, enabling an expressive but compact representation of motion and
semantics. To incorporate the learned representation in the second stage, we
utilize object-level attention objectives. Specifically, we consider a spatial,
per-object, masked-cross attention objective, integrating object-specific
prompts into corresponding latent space regions and a masked spatio-temporal
self-attention objective, ensuring frame-to-frame consistency for each object.
We evaluate our method on challenging benchmarks with multi-object and
high-motion scenarios and empirically demonstrate that the proposed method
achieves state-of-the-art results in temporal coherence, motion realism, and
text-prompt faithfulness. Additionally, we introduce \benchmark, a new
challenging benchmark for single-object and multi-object I2V generation, and
demonstrate our method's superiority on this benchmark. Project page is
available at https://guyyariv.github.io/TTM/.
677cd67175191fdc79cbcc04
Guy Yariv
claimed_verified
2025-01-07T09:37:35.128Z
646d239f4220471ca0c6471c
https://cdn-avatars.huggingface.co/v1/production/uploads/646d239f4220471ca0c6471c/sRwzko8XEUVCkeD7jXceH.jpeg
Guy Yariv
user
GuyYariv
677cd67175191fdc79cbcc05
Yuval Kirstain
677cd67175191fdc79cbcc06
Amit Zohar
admin_assigned
2025-01-07T13:17:40.162Z
64d8cb34505306fcd2fb89c3
/avatars/4ae9ffe5d494a7d53bc519ff8e402b3e.svg
Amit Zohar
user
amitz
677cd67175191fdc79cbcc07
Shelly Sheynin
admin_assigned
2025-01-07T13:17:46.692Z
63468e6ca6f101c7f9049132
https://cdn-avatars.huggingface.co/v1/production/uploads/1665662111525-63468e6ca6f101c7f9049132.png
Shelly Sheynin
user
shellysheynin
677cd67175191fdc79cbcc08
Yaniv Taigman
677cd67175191fdc79cbcc09
Yossi Adi
admin_assigned
2025-01-07T13:17:56.992Z
6481e135578646b5c2386728
https://cdn-avatars.huggingface.co/v1/production/uploads/6481e135578646b5c2386728/SPva4iNw0pORiCXD45cx9.jpeg
Yossi Adi
user
adiyoss
677cd67175191fdc79cbcc0a
Sagie Benaim
admin_assigned
2025-01-07T13:18:03.458Z
6345a9b9a8c2ff9f1377faab
/avatars/a5f2b999ef8b967b2af9f41afcd9d475.svg
Sagie Benaim
user
sagiebenaim
677cd67175191fdc79cbcc0b
Adam Polyak
admin_assigned
2025-01-07T13:18:10.411Z
6304d514dae2eb7d08413d62
/avatars/02a571bc791b78d3993d9a0484b70a29.svg
Adam Polyak
user
adampo
Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video
  Generation
We consider the task of Image-to-Video (I2V) generation, which involves
transforming static images into realistic video sequences based on a textual
description. While recent advancements produce photorealistic outputs, they
frequently struggle to create videos with accurate and consistent object
motion, especially in multi-object scenarios. To address these limitations, we
propose a two-stage compositional framework that decomposes I2V generation
into: (i) An explicit intermediate representation generation stage, followed by
(ii) A video generation stage that is conditioned on this representation. Our
key innovation is the introduction of a mask-based motion trajectory as an
intermediate representation, that captures both semantic object information and
motion, enabling an expressive but compact representation of motion and
semantics. To incorporate the learned representation in the second stage, we
utilize object-level attention objectives. Specifically, we consider a spatial,
per-object, masked-cross attention objective, integrating object-specific
prompts into corresponding latent space regions and a masked spatio-temporal
self-attention objective, ensuring frame-to-frame consistency for each object.
We evaluate our method on challenging benchmarks with multi-object and
high-motion scenarios and empirically demonstrate that the proposed method
achieves state-of-the-art results in temporal coherence, motion realism, and
text-prompt faithfulness. Additionally, we introduce \benchmark, a new
challenging benchmark for single-object and multi-object I2V generation, and
demonstrate our method's superiority on this benchmark. Project page is
available at https://guyyariv.github.io/TTM/.
The proposed two-stage framework for Image-to-Video generation uses mask-based motion trajectories and object-level attention to produce realistic and semantically consistent videos from text descriptions.
image-to-video generation intermediate representation mask-based motion trajectory object-level attention masked-cross attention masked spatio-temporal self-attention temporal coherence motion realism text-prompt faithfulness