EdiVal-Agent: An Object-Centric Framework for Automated, Scalable,
  Fine-Grained Evaluation of Multi-Turn Editing
Instruction-based image editing has advanced rapidly, yet reliable and
interpretable evaluation remains a bottleneck. Current protocols either (i)
depend on paired reference images -- resulting in limited coverage and
inheriting biases from prior generative models -- or (ii) rely solely on
zero-shot vision-language models (VLMs), whose prompt-based assessments of
instruction following, content consistency, and visual quality are often
imprecise.
  To address this, we introduce EdiVal-Agent, an automated, scalable, and
fine-grained evaluation framework for multi-turn instruction-based editing from
an object-centric perspective, supported by a suite of expert tools. Given an
image, EdiVal-Agent first decomposes it into semantically meaningful objects,
then synthesizes diverse, context-aware editing instructions. For evaluation,
it integrates VLMs with open-vocabulary object detectors to assess instruction
following, uses semantic-level feature extractors to evaluate content
consistency, and leverages human preference models to judge visual quality. We
show that combining VLMs with object detectors yields stronger agreement with
human judgments in instruction-following evaluation compared to using VLMs
alone and CLIP-based metrics. Furthermore, the pipeline's modular design allows
future tools to be seamlessly integrated, enhancing evaluation accuracy over
time.
  Instantiating this pipeline, we build EdiVal-Bench, a multi-turn editing
benchmark covering 9 instruction types and 11 state-of-the-art editing models
spanning autoregressive (AR) (including Nano Banana, GPT-Image-1),
flow-matching, and diffusion paradigms. We demonstrate that EdiVal-Agent can be
used to identify existing failure modes, thereby informing the development of
the next generation of editing models. Project page:
https://tianyucodings.github.io/EdiVAL-page/.
68ccff113df9ac65e93dc734
Tianyu Chen
claimed_verified
2025-09-19T10:25:19.099Z
6630a1da64b627c0b9a57360
/avatars/9070ca03d2f7eef1e7e243c14d89274f.svg
Tianyu Chen
user
C-Tianyu
68ccff113df9ac65e93dc735
Yasi Zhang
68ccff113df9ac65e93dc736
Zhi Zhang
68ccff113df9ac65e93dc737
Peiyu Yu
68ccff113df9ac65e93dc738
Shu Wang
68ccff113df9ac65e93dc739
Zhendong Wang
68ccff113df9ac65e93dc73a
Kevin Lin
68ccff113df9ac65e93dc73b
Xiaofei Wang
68ccff113df9ac65e93dc73c
Zhengyuan Yang
68ccff113df9ac65e93dc73d
Linjie Li
68ccff113df9ac65e93dc73e
Chung-Ching Lin
68ccff113df9ac65e93dc73f
Jianwen Xie
68ccff113df9ac65e93dc740
Oscar Leong
68ccff113df9ac65e93dc741
Lijuan Wang
68ccff113df9ac65e93dc742
Ying Nian Wu
68ccff113df9ac65e93dc743
Mingyuan Zhou
EdiVal-Agent: An Object-Centric Framework for Automated, Scalable,
  Fine-Grained Evaluation of Multi-Turn Editing
Instruction-based image editing has advanced rapidly, yet reliable and
interpretable evaluation remains a bottleneck. Current protocols either (i)
depend on paired reference images -- resulting in limited coverage and
inheriting biases from prior generative models -- or (ii) rely solely on
zero-shot vision-language models (VLMs), whose prompt-based assessments of
instruction following, content consistency, and visual quality are often
imprecise.
  To address this, we introduce EdiVal-Agent, an automated, scalable, and
fine-grained evaluation framework for multi-turn instruction-based editing from
an object-centric perspective, supported by a suite of expert tools. Given an
image, EdiVal-Agent first decomposes it into semantically meaningful objects,
then synthesizes diverse, context-aware editing instructions. For evaluation,
it integrates VLMs with open-vocabulary object detectors to assess instruction
following, uses semantic-level feature extractors to evaluate content
consistency, and leverages human preference models to judge visual quality. We
show that combining VLMs with object detectors yields stronger agreement with
human judgments in instruction-following evaluation compared to using VLMs
alone and CLIP-based metrics. Furthermore, the pipeline's modular design allows
future tools to be seamlessly integrated, enhancing evaluation accuracy over
time.
  Instantiating this pipeline, we build EdiVal-Bench, a multi-turn editing
benchmark covering 9 instruction types and 11 state-of-the-art editing models
spanning autoregressive (AR) (including Nano Banana, GPT-Image-1),
flow-matching, and diffusion paradigms. We demonstrate that EdiVal-Agent can be
used to identify existing failure modes, thereby informing the development of
the next generation of editing models. Project page:
https://tianyucodings.github.io/EdiVAL-page/.
EdiVal-Agent is an automated evaluation framework for instruction-based image editing that integrates VLMs, object detectors, and human preference models to assess instruction following, content consistency, and visual quality.
EdiVal-Agent zero-shot vision-language models VLMs open-vocabulary object detectors semantic-level feature extractors human preference models EdiVal-Bench autoregressive flow-matching diffusion paradigms