MagicStick: Controllable Video Editing via Control Handle
  Transformations
Text-based video editing has recently attracted considerable interest in
changing the style or replacing the objects with a similar structure. Beyond
this, we demonstrate that properties such as shape, size, location, motion,
etc., can also be edited in videos. Our key insight is that the keyframe
transformations of the specific internal feature (e.g., edge maps of objects or
human pose), can easily propagate to other frames to provide generation
guidance. We thus propose MagicStick, a controllable video editing method that
edits the video properties by utilizing the transformation on the extracted
internal control signals. In detail, to keep the appearance, we inflate both
the pretrained image diffusion model and ControlNet to the temporal dimension
and train low-rank adaptions (LORA) layers to fit the specific scenes. Then, in
editing, we perform an inversion and editing framework. Differently, finetuned
ControlNet is introduced in both inversion and generation for attention
guidance with the proposed attention remix between the spatial attention maps
of inversion and editing. Yet succinct, our method is the first method to show
the ability of video property editing from the pre-trained text-to-image model.
We present experiments on numerous examples within our unified framework. We
also compare with shape-aware text-based editing and handcrafted motion video
generation, demonstrating our superior temporal consistency and editing
capability than previous works. The code and models will be made publicly
available.
65713489c8018fe6408d9ba4
Yue Ma
65713489c8018fe6408d9ba5
Xiaodong Cun
admin_assigned
2023-12-07T12:00:42.458Z
63184c517ca1b876d99b7e0e
https://cdn-avatars.huggingface.co/v1/production/uploads/63184c517ca1b876d99b7e0e/b-qDExoeJuDXK0cJBZKnz.jpeg
Xiaodong Cun
user
vinthony
65713489c8018fe6408d9ba6
Yingqing He
admin_assigned
2023-12-07T12:00:52.831Z
630231bf7e137e3d6b3b0645
/avatars/dad52f8955110f0a2caeb613d6aa3ea2.svg
He
user
Yingqing
65713489c8018fe6408d9ba7
Chenyang Qi
admin_assigned
2023-12-07T12:01:03.648Z
642157a201c62c1e41fc526a
/avatars/73ae831a73947a04468b1e87d17da24d.svg
chenyangqi
user
chenyangqi
65713489c8018fe6408d9ba8
Xintao Wang
admin_assigned
2023-12-07T12:01:13.305Z
60e272ca6c78a8c122b12127
https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
Xintao Wang
user
Xintao
65713489c8018fe6408d9ba9
Ying Shan
65713489c8018fe6408d9baa
Xiu Li
65713489c8018fe6408d9bab
Qifeng Chen
admin_assigned
2023-12-07T12:01:44.912Z
6467b121e7a6a374fd19b44b
/avatars/3f2874d58986d651aef55e3408b05700.svg
Qifeng Chen
user
cqf
MagicStick: Controllable Video Editing via Control Handle
  Transformations
Text-based video editing has recently attracted considerable interest in
changing the style or replacing the objects with a similar structure. Beyond
this, we demonstrate that properties such as shape, size, location, motion,
etc., can also be edited in videos. Our key insight is that the keyframe
transformations of the specific internal feature (e.g., edge maps of objects or
human pose), can easily propagate to other frames to provide generation
guidance. We thus propose MagicStick, a controllable video editing method that
edits the video properties by utilizing the transformation on the extracted
internal control signals. In detail, to keep the appearance, we inflate both
the pretrained image diffusion model and ControlNet to the temporal dimension
and train low-rank adaptions (LORA) layers to fit the specific scenes. Then, in
editing, we perform an inversion and editing framework. Differently, finetuned
ControlNet is introduced in both inversion and generation for attention
guidance with the proposed attention remix between the spatial attention maps
of inversion and editing. Yet succinct, our method is the first method to show
the ability of video property editing from the pre-trained text-to-image model.
We present experiments on numerous examples within our unified framework. We
also compare with shape-aware text-based editing and handcrafted motion video
generation, demonstrating our superior temporal consistency and editing
capability than previous works. The code and models will be made publicly
available.
MagicStick, a text-based video editing method, uses internal feature transformations to edit video properties, achieving superior temporal consistency and editing capability by integrating pretrained image diffusion models and ControlNet with attention remix.
diffusion models ControlNet low-rank adaptions LORA attention remix temporal consistency text-to-image model