FramePainter: Endowing Interactive Image Editing with Video Diffusion
  Priors
Interactive image editing allows users to modify images through visual
interaction operations such as drawing, clicking, and dragging. Existing
methods construct such supervision signals from videos, as they capture how
objects change with various physical interactions. However, these models are
usually built upon text-to-image diffusion models, so necessitate (i) massive
training samples and (ii) an additional reference encoder to learn real-world
dynamics and visual consistency. In this paper, we reformulate this task as an
image-to-video generation problem, so that inherit powerful video diffusion
priors to reduce training costs and ensure temporal consistency. Specifically,
we introduce FramePainter as an efficient instantiation of this formulation.
Initialized with Stable Video Diffusion, it only uses a lightweight sparse
control encoder to inject editing signals. Considering the limitations of
temporal attention in handling large motion between two frames, we further
propose matching attention to enlarge the receptive field while encouraging
dense correspondence between edited and source image tokens. We highlight the
effectiveness and efficiency of FramePainter across various of editing signals:
it domainantly outperforms previous state-of-the-art methods with far less
training data, achieving highly seamless and coherent editing of images, \eg,
automatically adjust the reflection of the cup. Moreover, FramePainter also
exhibits exceptional generalization in scenarios not present in real-world
videos, \eg, transform the clownfish into shark-like shape. Our code will be
available at https://github.com/YBYBZhang/FramePainter.
67872d3659aa2a284c52ad5f
Yabo Zhang
admin_assigned
2025-01-15T16:48:10.703Z
62cd752ba9be5c19555c2b4c
/avatars/ed72684f0f2139516ccde24cd467cea6.svg
YaboZhang
user
Yabo
67872d3659aa2a284c52ad60
Xinpeng Zhou
admin_assigned
2025-01-15T16:48:17.107Z
66a7044a315d9b5c32d2a399
/avatars/e2da8af4a4f758aa5de7bf34b085f764.svg
zhou
user
xinpengzhou
67872d3659aa2a284c52ad61
Yihan Zeng
admin_assigned
2025-01-15T16:48:26.078Z
65beb318358734fd090f742e
/avatars/962420a21b2aa8689dd0e1d4531fbf35.svg
Yihan Zeng
user
vikyzeng2
67872d3659aa2a284c52ad62
Hang Xu
67872d3659aa2a284c52ad63
Hui Li
67872d3659aa2a284c52ad64
Wangmeng Zuo
FramePainter: Endowing Interactive Image Editing with Video Diffusion
  Priors
Interactive image editing allows users to modify images through visual
interaction operations such as drawing, clicking, and dragging. Existing
methods construct such supervision signals from videos, as they capture how
objects change with various physical interactions. However, these models are
usually built upon text-to-image diffusion models, so necessitate (i) massive
training samples and (ii) an additional reference encoder to learn real-world
dynamics and visual consistency. In this paper, we reformulate this task as an
image-to-video generation problem, so that inherit powerful video diffusion
priors to reduce training costs and ensure temporal consistency. Specifically,
we introduce FramePainter as an efficient instantiation of this formulation.
Initialized with Stable Video Diffusion, it only uses a lightweight sparse
control encoder to inject editing signals. Considering the limitations of
temporal attention in handling large motion between two frames, we further
propose matching attention to enlarge the receptive field while encouraging
dense correspondence between edited and source image tokens. We highlight the
effectiveness and efficiency of FramePainter across various of editing signals:
it domainantly outperforms previous state-of-the-art methods with far less
training data, achieving highly seamless and coherent editing of images, \eg,
automatically adjust the reflection of the cup. Moreover, FramePainter also
exhibits exceptional generalization in scenarios not present in real-world
videos, \eg, transform the clownfish into shark-like shape. Our code will be
available at https://github.com/YBYBZhang/FramePainter.
FramePainter, an image-to-video generation model, efficiently handles interactive image editing with sparse control and matching attention for seamless and coherent edits.
image-to-video generation video diffusion priors FramePainter Stable Video Diffusion sparse control encoder matching attention temporal attention dense correspondence image editing real-world dynamics visual consistency