BrushEdit: All-In-One Image Inpainting and Editing
Image editing has advanced significantly with the development of diffusion
models using both inversion-based and instruction-based methods. However,
current inversion-based approaches struggle with big modifications (e.g.,
adding or removing objects) due to the structured nature of inversion noise,
which hinders substantial changes. Meanwhile, instruction-based methods often
constrain users to black-box operations, limiting direct interaction for
specifying editing regions and intensity. To address these limitations, we
propose BrushEdit, a novel inpainting-based instruction-guided image editing
paradigm, which leverages multimodal large language models (MLLMs) and image
inpainting models to enable autonomous, user-friendly, and interactive
free-form instruction editing. Specifically, we devise a system enabling
free-form instruction editing by integrating MLLMs and a dual-branch image
inpainting model in an agent-cooperative framework to perform editing category
classification, main object identification, mask acquisition, and editing area
inpainting. Extensive experiments show that our framework effectively combines
MLLMs and inpainting models, achieving superior performance across seven
metrics including mask region preservation and editing effect coherence.
676044eedd3e9eeed65d9a23
Yaowei Li
admin_assigned
2024-12-17T09:41:12.208Z
6362801380c1a705a6ea54ac
/avatars/041ad5abf9be42e336938f51ebb8746c.svg
Yaowei Li
user
Yw22
676044eedd3e9eeed65d9a24
Yuxuan Bian
admin_assigned
2024-12-17T09:41:18.245Z
650447dd52ca06fef957f05d
/avatars/511c11ac9b3cc7a162bda5e07f6ee0a3.svg
Yuxuan BIAN
user
BianYx
676044eedd3e9eeed65d9a25
Xuan Ju
admin_assigned
2024-12-17T09:41:23.731Z
62d4577bc85b0fcf7fde39bb
/avatars/a3a5729e33ae89ce9ba408830db3c835.svg
Xuan Ju
user
juxuan27
676044eedd3e9eeed65d9a26
Zhaoyang Zhang
claimed_verified
2024-12-17T08:04:08.717Z
658409ceca19ccf6d9989add
/avatars/3ac1dbd25e52435185babdeb3da28875.svg
Zhaoyang Zhang
user
ZyZcuhk
676044eedd3e9eeed65d9a27
Ying Shan
admin_assigned
2024-12-17T09:41:29.223Z
63ca3ddc04c979828310bfcb
/avatars/615e0d8622950b4408b40d550f02a894.svg
Ying Shan
user
yshan2u
676044eedd3e9eeed65d9a28
Qiang Xu
BrushEdit: All-In-One Image Inpainting and Editing
Image editing has advanced significantly with the development of diffusion
models using both inversion-based and instruction-based methods. However,
current inversion-based approaches struggle with big modifications (e.g.,
adding or removing objects) due to the structured nature of inversion noise,
which hinders substantial changes. Meanwhile, instruction-based methods often
constrain users to black-box operations, limiting direct interaction for
specifying editing regions and intensity. To address these limitations, we
propose BrushEdit, a novel inpainting-based instruction-guided image editing
paradigm, which leverages multimodal large language models (MLLMs) and image
inpainting models to enable autonomous, user-friendly, and interactive
free-form instruction editing. Specifically, we devise a system enabling
free-form instruction editing by integrating MLLMs and a dual-branch image
inpainting model in an agent-cooperative framework to perform editing category
classification, main object identification, mask acquisition, and editing area
inpainting. Extensive experiments show that our framework effectively combines
MLLMs and inpainting models, achieving superior performance across seven
metrics including mask region preservation and editing effect coherence.
BrushEdit addresses image editing limitations by combining multimodal large language models and dual-branch inpainting models for autonomous and interactive free-form instruction editing.
diffusion models MLLMs multimodal large language models dual-branch image inpainting model agent-cooperative framework editing category classification main object identification mask acquisition editing area inpainting mask region preservation editing effect coherence