Image Editing As Programs with Diffusion Models
While diffusion models have achieved remarkable success in text-to-image
generation, they encounter significant challenges with instruction-driven image
editing. Our research highlights a key challenge: these models particularly
struggle with structurally inconsistent edits that involve substantial layout
changes. To mitigate this gap, we introduce Image Editing As Programs (IEAP), a
unified image editing framework built upon the Diffusion Transformer (DiT)
architecture. At its core, IEAP approaches instructional editing through a
reductionist lens, decomposing complex editing instructions into sequences of
atomic operations. Each operation is implemented via a lightweight adapter
sharing the same DiT backbone and is specialized for a specific type of edit.
Programmed by a vision-language model (VLM)-based agent, these operations
collaboratively support arbitrary and structurally inconsistent
transformations. By modularizing and sequencing edits in this way, IEAP
generalizes robustly across a wide range of editing tasks, from simple
adjustments to substantial structural changes. Extensive experiments
demonstrate that IEAP significantly outperforms state-of-the-art methods on
standard benchmarks across various editing scenarios. In these evaluations, our
framework delivers superior accuracy and semantic fidelity, particularly for
complex, multi-step instructions. Codes are available at
https://github.com/YujiaHu1109/IEAP.
6840fb71d4e16ff5f95108aa
Yujia Hu
claimed_verified
2025-06-05T12:42:05.838Z
67da3534e6387c9c9bc954b3
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/1IiNF_DgWQcvMYIlW8ikC.png
HU YUJIA
user
Cicici1109
6840fb71d4e16ff5f95108ab
Songhua Liu
6840fb71d4e16ff5f95108ac
Zhenxiong Tan
6840fb71d4e16ff5f95108ad
Xingyi Yang
claimed_verified
2025-06-05T07:27:11.256Z
634cfebc350bcee9bed20a4d
https://cdn-avatars.huggingface.co/v1/production/uploads/634cfebc350bcee9bed20a4d/fN47nN5rhw-HJaFLBZWQy.png
Xingyi Yang
user
adamdad
6840fb71d4e16ff5f95108ae
Xinchao Wang
Image Editing As Programs with Diffusion Models
While diffusion models have achieved remarkable success in text-to-image
generation, they encounter significant challenges with instruction-driven image
editing. Our research highlights a key challenge: these models particularly
struggle with structurally inconsistent edits that involve substantial layout
changes. To mitigate this gap, we introduce Image Editing As Programs (IEAP), a
unified image editing framework built upon the Diffusion Transformer (DiT)
architecture. At its core, IEAP approaches instructional editing through a
reductionist lens, decomposing complex editing instructions into sequences of
atomic operations. Each operation is implemented via a lightweight adapter
sharing the same DiT backbone and is specialized for a specific type of edit.
Programmed by a vision-language model (VLM)-based agent, these operations
collaboratively support arbitrary and structurally inconsistent
transformations. By modularizing and sequencing edits in this way, IEAP
generalizes robustly across a wide range of editing tasks, from simple
adjustments to substantial structural changes. Extensive experiments
demonstrate that IEAP significantly outperforms state-of-the-art methods on
standard benchmarks across various editing scenarios. In these evaluations, our
framework delivers superior accuracy and semantic fidelity, particularly for
complex, multi-step instructions. Codes are available at
https://github.com/YujiaHu1109/IEAP.
A unified image editing framework, IEAP, built on Diffusion Transformer (DiT) decomposes complex editing instructions into operations performed by vision-language models for robust editing across various tasks.
diffusion models text-to-image generation instruction-driven image editing structurally inconsistent edits Image Editing As Programs (IEAP) Diffusion Transformer (DiT) atomic operations lightweight adapter vision-language model (VLM) modularizing edits