GenCompositor: Generative Video Compositing with Diffusion Transformer
Video compositing combines live-action footage to create video production,
serving as a crucial technique in video creation and film production.
Traditional pipelines require intensive labor efforts and expert collaboration,
resulting in lengthy production cycles and high manpower costs. To address this
issue, we automate this process with generative models, called generative video
compositing. This new task strives to adaptively inject identity and motion
information of foreground video to the target video in an interactive manner,
allowing users to customize the size, motion trajectory, and other attributes
of the dynamic elements added in final video. Specifically, we designed a novel
Diffusion Transformer (DiT) pipeline based on its intrinsic properties. To
maintain consistency of the target video before and after editing, we revised a
light-weight DiT-based background preservation branch with masked token
injection. As to inherit dynamic elements from other sources, a DiT fusion
block is proposed using full self-attention, along with a simple yet effective
foreground augmentation for training. Besides, for fusing background and
foreground videos with different layouts based on user control, we developed a
novel position embedding, named Extended Rotary Position Embedding (ERoPE).
Finally, we curated a dataset comprising 61K sets of videos for our new task,
called VideoComp. This data includes complete dynamic elements and high-quality
target videos. Experiments demonstrate that our method effectively realizes
generative video compositing, outperforming existing possible solutions in
fidelity and consistency.
68b7e0a3295f15ff609114ba
Shuzhou Yang
claimed_verified
2025-09-03T08:26:00.961Z
637db2dbb61b6d662af326e8
/avatars/6d543ec2847755e95943041f695634b9.svg
Shuzhou Yang
user
Ysz2022
68b7e0a3295f15ff609114bb
Xiaoyu Li
claimed_verified
2025-09-03T09:06:28.411Z
66d8284b6bddfb32e77ddafb
/avatars/edd371ecef6e7d58b945a30bbc6095ee.svg
Xiaoyu Li
user
Xiaoyu521
68b7e0a3295f15ff609114bc
Xiaodong Cun
claimed_verified
2025-09-09T13:49:54.054Z
63184c517ca1b876d99b7e0e
https://cdn-avatars.huggingface.co/v1/production/uploads/63184c517ca1b876d99b7e0e/b-qDExoeJuDXK0cJBZKnz.jpeg
Xiaodong Cun
user
vinthony
68b7e0a3295f15ff609114bd
Guangzhi Wang
68b7e0a3295f15ff609114be
Lingen Li
68b7e0a3295f15ff609114bf
Ying Shan
68b7e0a3295f15ff609114c0
Jian Zhang
GenCompositor: Generative Video Compositing with Diffusion Transformer
Video compositing combines live-action footage to create video production,
serving as a crucial technique in video creation and film production.
Traditional pipelines require intensive labor efforts and expert collaboration,
resulting in lengthy production cycles and high manpower costs. To address this
issue, we automate this process with generative models, called generative video
compositing. This new task strives to adaptively inject identity and motion
information of foreground video to the target video in an interactive manner,
allowing users to customize the size, motion trajectory, and other attributes
of the dynamic elements added in final video. Specifically, we designed a novel
Diffusion Transformer (DiT) pipeline based on its intrinsic properties. To
maintain consistency of the target video before and after editing, we revised a
light-weight DiT-based background preservation branch with masked token
injection. As to inherit dynamic elements from other sources, a DiT fusion
block is proposed using full self-attention, along with a simple yet effective
foreground augmentation for training. Besides, for fusing background and
foreground videos with different layouts based on user control, we developed a
novel position embedding, named Extended Rotary Position Embedding (ERoPE).
Finally, we curated a dataset comprising 61K sets of videos for our new task,
called VideoComp. This data includes complete dynamic elements and high-quality
target videos. Experiments demonstrate that our method effectively realizes
generative video compositing, outperforming existing possible solutions in
fidelity and consistency.
A novel Diffusion Transformer pipeline automates video compositing by adaptively injecting identity and motion information, maintaining consistency and enabling user customization.
Diffusion Transformer DiT masked token injection full self-attention foreground augmentation Extended Rotary Position Embedding ERoPE generative video compositing