VideoSwap: Customized Video Subject Swapping with Interactive Semantic
  Point Correspondence
Current diffusion-based video editing primarily focuses on
structure-preserved editing by utilizing various dense correspondences to
ensure temporal consistency and motion alignment. However, these approaches are
often ineffective when the target edit involves a shape change. To embark on
video editing with shape change, we explore customized video subject swapping
in this work, where we aim to replace the main subject in a source video with a
target subject having a distinct identity and potentially different shape. In
contrast to previous methods that rely on dense correspondences, we introduce
the VideoSwap framework that exploits semantic point correspondences, inspired
by our observation that only a small number of semantic points are necessary to
align the subject's motion trajectory and modify its shape. We also introduce
various user-point interactions (\eg, removing points and dragging points) to
address various semantic point correspondence. Extensive experiments
demonstrate state-of-the-art video subject swapping results across a variety of
real-world videos.
656eb2b28e7f2775acfc13d8
Yuchao Gu
admin_assigned
2023-12-05T09:25:39.377Z
63021630a35b21bd8a53305a
/avatars/7a7e8b39749eda61e57d8a1908726558.svg
Gu Yuchao
user
guyuchao
656eb2b28e7f2775acfc13d9
Yipin Zhou
656eb2b28e7f2775acfc13da
Bichen Wu
admin_assigned
2023-12-05T09:26:04.012Z
6363749520c2cf37bccd9e84
/avatars/e9de7911e67090d883f15a7a58f18e4e.svg
Bichen Wu
user
bichenwu
656eb2b28e7f2775acfc13db
Licheng Yu
admin_assigned
2023-12-05T09:26:11.614Z
6372b1f9bd81fae2b3a712e8
/avatars/bcdf7b540cc94882e0ae6035bb07a8f2.svg
Licheng Yu
user
lichengyu
656eb2b28e7f2775acfc13dc
Jia-Wei Liu
admin_assigned
2023-12-05T09:26:35.038Z
64230048a73327caad9d8241
/avatars/ac115f1a21c743db7c925f0e18451145.svg
Jiawei Liu
user
JiaweiLIU
656eb2b28e7f2775acfc13dd
Rui Zhao
admin_assigned
2023-12-05T09:26:59.193Z
652b83b73b5997ed71a310f2
https://cdn-avatars.huggingface.co/v1/production/uploads/652b83b73b5997ed71a310f2/ipCpdeHUp4-0OmRz5z8IW.png
Rui Zhao
user
ruizhaocv
656eb2b28e7f2775acfc13de
Jay Zhangjie Wu
claimed_verified
2023-12-06T10:25:09.481Z
633aaf695df91da9cea92960
https://cdn-avatars.huggingface.co/v1/production/uploads/633aaf695df91da9cea92960/h1oOWUudbdlsd63Q5w0hM.png
Jay Wu
user
jayw
656eb2b28e7f2775acfc13df
David Junhao Zhang
admin_assigned
2023-12-05T09:27:34.361Z
6468855763a564ba347e1efa
/avatars/caa99c3318c1d6d21be44d2d32795a62.svg
David Junhao ZHANG
user
Junhao233
656eb2b28e7f2775acfc13e0
Mike Zheng Shou
admin_assigned
2023-12-05T09:28:06.371Z
63a55320ce5763e06f78519c
https://cdn-avatars.huggingface.co/v1/production/uploads/1671779060549-noauth.jpeg
Mike Shou
user
mikeshou
656eb2b28e7f2775acfc13e1
Kevin Tang
VideoSwap: Customized Video Subject Swapping with Interactive Semantic
  Point Correspondence
Current diffusion-based video editing primarily focuses on
structure-preserved editing by utilizing various dense correspondences to
ensure temporal consistency and motion alignment. However, these approaches are
often ineffective when the target edit involves a shape change. To embark on
video editing with shape change, we explore customized video subject swapping
in this work, where we aim to replace the main subject in a source video with a
target subject having a distinct identity and potentially different shape. In
contrast to previous methods that rely on dense correspondences, we introduce
the VideoSwap framework that exploits semantic point correspondences, inspired
by our observation that only a small number of semantic points are necessary to
align the subject's motion trajectory and modify its shape. We also introduce
various user-point interactions (\eg, removing points and dragging points) to
address various semantic point correspondence. Extensive experiments
demonstrate state-of-the-art video subject swapping results across a variety of
real-world videos.
The VideoSwap framework uses semantic point correspondences for video subject swapping, achieving superior results in handling shape changes compared to dense correspondence methods.
diffusion-based video editing structure-preserved editing dense correspondences VideoSwap semantic point correspondences user-point interactions