OmniEdit: Building Image Editing Generalist Models Through Specialist
  Supervision
Instruction-guided image editing methods have demonstrated significant
potential by training diffusion models on automatically synthesized or manually
annotated image editing pairs. However, these methods remain far from
practical, real-life applications. We identify three primary challenges
contributing to this gap. Firstly, existing models have limited editing skills
due to the biased synthesis process. Secondly, these methods are trained with
datasets with a high volume of noise and artifacts. This is due to the
application of simple filtering methods like CLIP-score. Thirdly, all these
datasets are restricted to a single low resolution and fixed aspect ratio,
limiting the versatility to handle real-world use cases. In this paper, we
present \omniedit, which is an omnipotent editor to handle seven different
image editing tasks with any aspect ratio seamlessly. Our contribution is in
four folds: (1) \omniedit is trained by utilizing the supervision from seven
different specialist models to ensure task coverage. (2) we utilize importance
sampling based on the scores provided by large multimodal models (like GPT-4o)
instead of CLIP-score to improve the data quality. (3) we propose a new editing
architecture called EditNet to greatly boost the editing success rate, (4) we
provide images with different aspect ratios to ensure that our model can handle
any image in the wild. We have curated a test set containing images of
different aspect ratios, accompanied by diverse instructions to cover different
tasks. Both automatic evaluation and human evaluations demonstrate that
\omniedit can significantly outperform all the existing models. Our code,
dataset and model will be available at
https://tiger-ai-lab.github.io/OmniEdit/
6732d435f014ee495567a302
Cong Wei
claimed_verified
2024-11-15T09:26:52.427Z
64f8e358766ff9f3d2b0de84
https://cdn-avatars.huggingface.co/v1/production/uploads/64f8e358766ff9f3d2b0de84/R2P1YG-mRBh7TU9wkjGGk.jpeg
Cong Wei
user
CongWei1230
6732d435f014ee495567a303
Zheyang Xiong
claimed_verified
2024-11-15T09:27:00.558Z
67084dbb652deb3c09d8f633
/avatars/6489cc414ec6b7be3523e335ba14888f.svg
Zheyang Xiong
user
zheyang-hf
6732d435f014ee495567a304
Weiming Ren
claimed_verified
2024-11-29T21:10:08.592Z
64405a9d518271b0d1beea38
/avatars/b702474588fd7090773320422417a582.svg
Weiming Ren
user
wren93
6732d435f014ee495567a305
Xinrun Du
claimed_verified
2024-11-15T09:26:50.357Z
654907a4a1faff97850c4eff
/avatars/458c90151614bc7f116943b6e67d6b8a.svg
du
user
dododododo
6732d435f014ee495567a306
Ge Zhang
claimed_verified
2024-11-21T21:33:10.345Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
6732d435f014ee495567a307
Wenhu Chen
extracted_pending
2024-11-12T04:06:18.315Z
6313a86154e6e5d9f0f94e04
https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
Wenhu Chen
user
wenhu
OmniEdit: Building Image Editing Generalist Models Through Specialist
  Supervision
Instruction-guided image editing methods have demonstrated significant
potential by training diffusion models on automatically synthesized or manually
annotated image editing pairs. However, these methods remain far from
practical, real-life applications. We identify three primary challenges
contributing to this gap. Firstly, existing models have limited editing skills
due to the biased synthesis process. Secondly, these methods are trained with
datasets with a high volume of noise and artifacts. This is due to the
application of simple filtering methods like CLIP-score. Thirdly, all these
datasets are restricted to a single low resolution and fixed aspect ratio,
limiting the versatility to handle real-world use cases. In this paper, we
present \omniedit, which is an omnipotent editor to handle seven different
image editing tasks with any aspect ratio seamlessly. Our contribution is in
four folds: (1) \omniedit is trained by utilizing the supervision from seven
different specialist models to ensure task coverage. (2) we utilize importance
sampling based on the scores provided by large multimodal models (like GPT-4o)
instead of CLIP-score to improve the data quality. (3) we propose a new editing
architecture called EditNet to greatly boost the editing success rate, (4) we
provide images with different aspect ratios to ensure that our model can handle
any image in the wild. We have curated a test set containing images of
different aspect ratios, accompanied by diverse instructions to cover different
tasks. Both automatic evaluation and human evaluations demonstrate that
\omniedit can significantly outperform all the existing models. Our code,
dataset and model will be available at
https://tiger-ai-lab.github.io/OmniEdit/
An omnipotent image editor trained on multisource supervision, importance sampling, and a versatile architecture excels in multiple tasks with diverse aspect ratios and high data quality.
diffusion models CLIP-score GPT-4o EditNet aspect ratio image editing tasks automatic evaluation human evaluations