Zero-shot Image Editing with Reference Imitation
Image editing serves as a practical yet challenging task considering the
diverse demands from users, where one of the hardest parts is to precisely
describe how the edited image should look like. In this work, we present a new
form of editing, termed imitative editing, to help users exercise their
creativity more conveniently. Concretely, to edit an image region of interest,
users are free to directly draw inspiration from some in-the-wild references
(e.g., some relative pictures come across online), without having to cope with
the fit between the reference and the source. Such a design requires the system
to automatically figure out what to expect from the reference to perform the
editing. For this purpose, we propose a generative training framework, dubbed
MimicBrush, which randomly selects two frames from a video clip, masks some
regions of one frame, and learns to recover the masked regions using the
information from the other frame. That way, our model, developed from a
diffusion prior, is able to capture the semantic correspondence between
separate images in a self-supervised manner. We experimentally show the
effectiveness of our method under various test cases as well as its superiority
over existing alternatives. We also construct a benchmark to facilitate further
research.
6669102952faf47d18f0e07e
Xi Chen
claimed_verified
2024-06-12T07:29:29.878Z
644a1b6401e18bf93a6f45c1
https://cdn-avatars.huggingface.co/v1/production/uploads/644a1b6401e18bf93a6f45c1/P0i_CgCrIzOS2tYRlxoE9.png
xichen
user
xichenhku
6669102952faf47d18f0e07f
Yutong Feng
admin_assigned
2024-06-12T07:47:03.236Z
64a54e468cfaa458bd6844bf
https://cdn-avatars.huggingface.co/v1/production/uploads/64a54e468cfaa458bd6844bf/5Gmf4tAr59GNl-2VaZDbu.png
Yutong Feng
user
fengyutong
6669102952faf47d18f0e080
Mengting Chen
admin_assigned
2024-06-12T07:47:46.418Z
641073eb928400b416444af5
/avatars/0eed14a8929682f182e474f3693fc442.svg
Chenmengting
user
MengTingChen
6669102952faf47d18f0e081
Yiyang Wang
admin_assigned
2024-06-12T07:47:53.032Z
65664c8fcc78541029e4f4ac
/avatars/7182aa41dd4c7f74f7b566506543d33a.svg
Yiyang Wang
user
dwyy
6669102952faf47d18f0e082
Shilong Zhang
admin_assigned
2024-06-12T07:47:59.760Z
6424ffce46d202ad3d918a67
https://cdn-avatars.huggingface.co/v1/production/uploads/6424ffce46d202ad3d918a67/gmYmOA072fP_5cJLc9Qs4.jpeg
Shilong Zhang
user
shilongz
6669102952faf47d18f0e083
Yu Liu
admin_assigned
2024-06-12T07:48:29.016Z
636de85cc4a7a729c164d2b5
/avatars/3e281e547e1697e1c06805e7e63f3918.svg
Yu Liu
user
YuLiu
6669102952faf47d18f0e084
Yujun Shen
6669102952faf47d18f0e085
Hengshuang Zhao
Zero-shot Image Editing with Reference Imitation
Image editing serves as a practical yet challenging task considering the
diverse demands from users, where one of the hardest parts is to precisely
describe how the edited image should look like. In this work, we present a new
form of editing, termed imitative editing, to help users exercise their
creativity more conveniently. Concretely, to edit an image region of interest,
users are free to directly draw inspiration from some in-the-wild references
(e.g., some relative pictures come across online), without having to cope with
the fit between the reference and the source. Such a design requires the system
to automatically figure out what to expect from the reference to perform the
editing. For this purpose, we propose a generative training framework, dubbed
MimicBrush, which randomly selects two frames from a video clip, masks some
regions of one frame, and learns to recover the masked regions using the
information from the other frame. That way, our model, developed from a
diffusion prior, is able to capture the semantic correspondence between
separate images in a self-supervised manner. We experimentally show the
effectiveness of our method under various test cases as well as its superiority
over existing alternatives. We also construct a benchmark to facilitate further
research.
MimicBrush, a generative training framework developed from a diffusion prior, enables imitative image editing by leveraging video frames and masked regions to capture semantic correspondence and improve editing quality.
generative training framework MimicBrush diffusion prior video frames masked regions semantic correspondence imitative editing