OmnimatteZero: Training-free Real-time Omnimatte with Pre-trained Video
  Diffusion Models
Omnimatte aims to decompose a given video into semantically meaningful
layers, including the background and individual objects along with their
associated effects, such as shadows and reflections. Existing methods often
require extensive training or costly self-supervised optimization. In this
paper, we present OmnimatteZero, a training-free approach that leverages
off-the-shelf pre-trained video diffusion models for omnimatte. It can remove
objects from videos, extract individual object layers along with their effects,
and composite those objects onto new videos. We accomplish this by adapting
zero-shot image inpainting techniques for video object removal, a task they
fail to handle effectively out-of-the-box. We then show that self-attention
maps capture information about the object and its footprints and use them to
inpaint the object's effects, leaving a clean background. Additionally, through
simple latent arithmetic, object layers can be isolated and recombined
seamlessly with new video layers to produce new videos. Evaluations show that
OmnimatteZero not only achieves superior performance in terms of background
reconstruction but also sets a new record for the fastest Omnimatte approach,
achieving real-time performance with minimal frame runtime.
67e2706af6cf2764a534d4a5
Dvir Samuel
extracted_confirmed
2025-03-25T09:00:01.542Z
630f0d48982455e61cc4cc08
/avatars/eea6ed2e112e830effa98a4661c5474f.svg
Samuel
user
Dvir
67e2706af6cf2764a534d4a6
Matan Levy
admin_assigned
2025-03-25T09:25:56.024Z
66633be10875aaaa9153c963
/avatars/f47aaaf7b029ad3e99f49676a8f9a479.svg
Matan Levy
user
m98levy
67e2706af6cf2764a534d4a7
Nir Darshan
claimed_verified
2025-03-26T20:49:13.626Z
67e420bb2c327e08f6b07657
/avatars/d82b4dd21141b9eb92b2c3cff74f342d.svg
Nir Darshan
user
nirdrs
67e2706af6cf2764a534d4a8
Gal Chechik
admin_assigned
2025-03-25T09:26:05.156Z
6493393f357b252af72196c5
https://cdn-avatars.huggingface.co/v1/production/uploads/6493393f357b252af72196c5/EWSy18XRcMRa_4XMM3Fu-.jpeg
Gal Chechik
user
galchechik
67e2706af6cf2764a534d4a9
Rami Ben-Ari
admin_assigned
2025-03-25T09:26:11.853Z
64c5f22c2581696666ebed88
/avatars/e85cd2d82f16ec10cad2b63929b2f05a.svg
Rami Ben-Ari
user
ramiben
OmnimatteZero: Training-free Real-time Omnimatte with Pre-trained Video
  Diffusion Models
Omnimatte aims to decompose a given video into semantically meaningful
layers, including the background and individual objects along with their
associated effects, such as shadows and reflections. Existing methods often
require extensive training or costly self-supervised optimization. In this
paper, we present OmnimatteZero, a training-free approach that leverages
off-the-shelf pre-trained video diffusion models for omnimatte. It can remove
objects from videos, extract individual object layers along with their effects,
and composite those objects onto new videos. We accomplish this by adapting
zero-shot image inpainting techniques for video object removal, a task they
fail to handle effectively out-of-the-box. We then show that self-attention
maps capture information about the object and its footprints and use them to
inpaint the object's effects, leaving a clean background. Additionally, through
simple latent arithmetic, object layers can be isolated and recombined
seamlessly with new video layers to produce new videos. Evaluations show that
OmnimatteZero not only achieves superior performance in terms of background
reconstruction but also sets a new record for the fastest Omnimatte approach,
achieving real-time performance with minimal frame runtime.
OmnimatteZero leverages pre-trained video diffusion models and zero-shot image inpainting to efficiently decompose videos into meaningful layers and perform object removal and recombination with real-time performance.
video diffusion models zero-shot image inpainting self-attention maps latent arithmetic object removal video compositing