Learning Action and Reasoning-Centric Image Editing from Videos and
  Simulations
An image editing model should be able to perform diverse edits, ranging from
object replacement, changing attributes or style, to performing actions or
movement, which require many forms of reasoning. Current general
instruction-guided editing models have significant shortcomings with action and
reasoning-centric edits. Object, attribute or stylistic changes can be learned
from visually static datasets. On the other hand, high-quality data for action
and reasoning-centric edits is scarce and has to come from entirely different
sources that cover e.g. physical dynamics, temporality and spatial reasoning.
To this end, we meticulously curate the AURORA Dataset
(Action-Reasoning-Object-Attribute), a collection of high-quality training
data, human-annotated and curated from videos and simulation engines. We focus
on a key aspect of quality training data: triplets (source image, prompt,
target image) contain a single meaningful visual change described by the
prompt, i.e., truly minimal changes between source and target images. To
demonstrate the value of our dataset, we evaluate an AURORA-finetuned model on
a new expert-curated benchmark (AURORA-Bench) covering 8 diverse editing tasks.
Our model significantly outperforms previous editing models as judged by human
raters. For automatic evaluations, we find important flaws in previous metrics
and caution their use for semantically hard editing tasks. Instead, we propose
a new automatic metric that focuses on discriminative understanding. We hope
that our efforts : (1) curating a quality training dataset and an evaluation
benchmark, (2) developing critical evaluations, and (3) releasing a
state-of-the-art model, will fuel further progress on general image editing.
668d7a65963fdbcecc61f8fd
Benno Krojer
extracted_confirmed
2024-07-09T20:17:59.541Z
6270c58780d5f35f8dbe42be
/avatars/d4d6e5eadfe9b1f47bce1c66728b24fc.svg
Benno Krojer
user
BennoKrojer
668d7a65963fdbcecc61f8fe
Dheeraj Vattikonda
admin_assigned
2024-07-09T21:08:30.712Z
65fb7b1c876d4642e8340e95
/avatars/c258c4528315243b98c1e60ba398e98b.svg
Dheeraj Vattikonda
user
Dheeraj46329
668d7a65963fdbcecc61f8ff
Luis Lara
claimed_verified
2024-07-09T21:08:53.780Z
63e972f1ccae1fe5c6211759
https://cdn-avatars.huggingface.co/v1/production/uploads/63e972f1ccae1fe5c6211759/AfKPgMdAraUtvbtJpoHFY.jpeg
Luis Lara
user
ludolara
668d7a65963fdbcecc61f900
Varun Jampani
admin_assigned
2024-07-09T21:08:38.230Z
630e7cfe3fc17ffc50f6e835
/avatars/0b742ff094a09f9374fafcd97ab9e002.svg
Varun Jampani
user
varunjampani
668d7a65963fdbcecc61f901
Eva Portelance
668d7a65963fdbcecc61f902
Christopher Pal
668d7a65963fdbcecc61f903
Siva Reddy
Learning Action and Reasoning-Centric Image Editing from Videos and
  Simulations
An image editing model should be able to perform diverse edits, ranging from
object replacement, changing attributes or style, to performing actions or
movement, which require many forms of reasoning. Current general
instruction-guided editing models have significant shortcomings with action and
reasoning-centric edits. Object, attribute or stylistic changes can be learned
from visually static datasets. On the other hand, high-quality data for action
and reasoning-centric edits is scarce and has to come from entirely different
sources that cover e.g. physical dynamics, temporality and spatial reasoning.
To this end, we meticulously curate the AURORA Dataset
(Action-Reasoning-Object-Attribute), a collection of high-quality training
data, human-annotated and curated from videos and simulation engines. We focus
on a key aspect of quality training data: triplets (source image, prompt,
target image) contain a single meaningful visual change described by the
prompt, i.e., truly minimal changes between source and target images. To
demonstrate the value of our dataset, we evaluate an AURORA-finetuned model on
a new expert-curated benchmark (AURORA-Bench) covering 8 diverse editing tasks.
Our model significantly outperforms previous editing models as judged by human
raters. For automatic evaluations, we find important flaws in previous metrics
and caution their use for semantically hard editing tasks. Instead, we propose
a new automatic metric that focuses on discriminative understanding. We hope
that our efforts : (1) curating a quality training dataset and an evaluation
benchmark, (2) developing critical evaluations, and (3) releasing a
state-of-the-art model, will fuel further progress on general image editing.
AURORA Dataset and benchmark improve general image editing by focusing on action, reasoning, and minimal visual changes, outperforming previous models with better automatic metrics.
AURORA Dataset image editing object replacement attribute changes style changes action edits reasoning edits physical dynamics temporality spatial reasoning AURORA-Bench triplets source image prompt target image human-annotated expert-curated automatic evaluation discriminative understanding