Magic Fixup: Streamlining Photo Editing by Watching Dynamic Videos
We propose a generative model that, given a coarsely edited image,
synthesizes a photorealistic output that follows the prescribed layout. Our
method transfers fine details from the original image and preserves the
identity of its parts. Yet, it adapts it to the lighting and context defined by
the new layout. Our key insight is that videos are a powerful source of
supervision for this task: objects and camera motions provide many observations
of how the world changes with viewpoint, lighting, and physical interactions.
We construct an image dataset in which each sample is a pair of source and
target frames extracted from the same video at randomly chosen time intervals.
We warp the source frame toward the target using two motion models that mimic
the expected test-time user edits. We supervise our model to translate the
warped image into the ground truth, starting from a pretrained diffusion model.
Our model design explicitly enables fine detail transfer from the source frame
to the generated image, while closely following the user-specified layout. We
show that by using simple segmentations and coarse 2D manipulations, we can
synthesize a photorealistic edit faithful to the user's input while addressing
second-order effects like harmonizing the lighting and physical interactions
between edited objects.
65fba1cfbeb503364b7ccc83
Hadi Alzayer
admin_assigned
2024-03-21T11:46:11.494Z
648d554fba166fc822582b14
/avatars/93072cde421c9d7d25614f967a9aa45a.svg
Hadi Alzayer
user
HadiZayer
65fba1cfbeb503364b7ccc84
Zhihao Xia
admin_assigned
2024-03-21T11:46:24.364Z
64df0d5a1961da512204db2c
/avatars/e08625382ba8244b72d2e94a11ff325a.svg
xiazhihao
user
xiazhihao
65fba1cfbeb503364b7ccc85
Xuaner Zhang
65fba1cfbeb503364b7ccc86
Eli Shechtman
65fba1cfbeb503364b7ccc87
Jia-Bin Huang
admin_assigned
2024-03-21T11:46:44.108Z
641c139b73296f7ee256970c
/avatars/5a2550d95e686640242840ad3bd0e680.svg
Jiabin Huang
user
YellowAddice
65fba1cfbeb503364b7ccc88
Michael Gharbi
admin_assigned
2024-03-21T11:46:49.904Z
658161ad208369a9403ca9a1
/avatars/e2324ab2a712100681b0b8127305212b.svg
Michael Gharbi
user
mgharbi
Magic Fixup: Streamlining Photo Editing by Watching Dynamic Videos
We propose a generative model that, given a coarsely edited image,
synthesizes a photorealistic output that follows the prescribed layout. Our
method transfers fine details from the original image and preserves the
identity of its parts. Yet, it adapts it to the lighting and context defined by
the new layout. Our key insight is that videos are a powerful source of
supervision for this task: objects and camera motions provide many observations
of how the world changes with viewpoint, lighting, and physical interactions.
We construct an image dataset in which each sample is a pair of source and
target frames extracted from the same video at randomly chosen time intervals.
We warp the source frame toward the target using two motion models that mimic
the expected test-time user edits. We supervise our model to translate the
warped image into the ground truth, starting from a pretrained diffusion model.
Our model design explicitly enables fine detail transfer from the source frame
to the generated image, while closely following the user-specified layout. We
show that by using simple segmentations and coarse 2D manipulations, we can
synthesize a photorealistic edit faithful to the user's input while addressing
second-order effects like harmonizing the lighting and physical interactions
between edited objects.
A generative model utilizes video-based supervision to synthesize photorealistic edits from coarse layouts, transferring details and harmonizing lighting and interactions.
generative model photorealistic output prescribed layout fine details identity preservation lighting context motion models image dataset source frames target frames pretrained diffusion model fine detail transfer user-specified layout segmentation 2D manipulations second-order effects harmonizing lighting physical interactions