ACE: All-round Creator and Editor Following Instructions via Diffusion
  Transformer
Diffusion models have emerged as a powerful generative technology and have
been found to be applicable in various scenarios. Most existing foundational
diffusion models are primarily designed for text-guided visual generation and
do not support multi-modal conditions, which are essential for many visual
editing tasks. This limitation prevents these foundational diffusion models
from serving as a unified model in the field of visual generation, like GPT-4
in the natural language processing field. In this work, we propose ACE, an
All-round Creator and Editor, which achieves comparable performance compared to
those expert models in a wide range of visual generation tasks. To achieve this
goal, we first introduce a unified condition format termed Long-context
Condition Unit (LCU), and propose a novel Transformer-based diffusion model
that uses LCU as input, aiming for joint training across various generation and
editing tasks. Furthermore, we propose an efficient data collection approach to
address the issue of the absence of available training data. It involves
acquiring pairwise images with synthesis-based or clustering-based pipelines
and supplying these pairs with accurate textual instructions by leveraging a
fine-tuned multi-modal large language model. To comprehensively evaluate the
performance of our model, we establish a benchmark of manually annotated pairs
data across a variety of visual generation tasks. The extensive experimental
results demonstrate the superiority of our model in visual generation fields.
Thanks to the all-in-one capabilities of our model, we can easily build a
multi-modal chat system that responds to any interactive request for image
creation using a single model to serve as the backend, avoiding the cumbersome
pipeline typically employed in visual agents. Code and models will be available
on the project page: https://ali-vilab.github.io/ace-page/.
66fcb2fab3f1da2ed11a5be6
Zhen Han
claimed_verified
2024-10-02T16:09:31.178Z
647ffabf28b737d7b9462eb2
/avatars/210441fc6645d08b36ad43734108f914.svg
Zhen Han
user
hanzhn
66fcb2fab3f1da2ed11a5be7
Zeyinzi Jiang
admin_assigned
2024-10-02T09:07:09.512Z
643d278b482011f5f2bd0fae
/avatars/70b8a7ffbfa2a1c4b6f5ff5e2b96b7bf.svg
jiangzeyinzi
user
jiangzeyinzi
66fcb2fab3f1da2ed11a5be8
Yulin Pan
claimed_verified
2024-10-08T07:19:36.366Z
65a9e000cab1bfcc3af8cfe2
/avatars/ddd0062dbeb5eb976841cec8a63c64a6.svg
Yulin Pan
user
pan-yl
66fcb2fab3f1da2ed11a5be9
Jingfeng Zhang
admin_assigned
2024-10-02T09:06:31.815Z
66f0e0262aee3cb7e981bbac
/avatars/f8f1e70469b5e047dc6e0e9dec6c5bc1.svg
Jingfeng Zhang
user
jingfengzhang
66fcb2fab3f1da2ed11a5bea
Chaojie Mao
admin_assigned
2024-10-02T09:06:26.022Z
6458970cab9a44f42f620a80
/avatars/f9779b0621c931f922440fec95342444.svg
chaojie mao
user
chaojiemao
66fcb2fab3f1da2ed11a5beb
Chenwei Xie
admin_assigned
2024-10-02T09:06:19.596Z
66592c72f4124d863fd55574
/avatars/98f0d5e6ba3728e8a1164aa5188a3298.svg
Chenwei Xie
user
chenweix7
66fcb2fab3f1da2ed11a5bec
Yu Liu
66fcb2fab3f1da2ed11a5bed
Jingren Zhou
admin_assigned
2024-10-02T09:05:23.903Z
602f88f5e8149a962412a667
/avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
Zhou
user
Jingren
ACE: All-round Creator and Editor Following Instructions via Diffusion
  Transformer
Diffusion models have emerged as a powerful generative technology and have
been found to be applicable in various scenarios. Most existing foundational
diffusion models are primarily designed for text-guided visual generation and
do not support multi-modal conditions, which are essential for many visual
editing tasks. This limitation prevents these foundational diffusion models
from serving as a unified model in the field of visual generation, like GPT-4
in the natural language processing field. In this work, we propose ACE, an
All-round Creator and Editor, which achieves comparable performance compared to
those expert models in a wide range of visual generation tasks. To achieve this
goal, we first introduce a unified condition format termed Long-context
Condition Unit (LCU), and propose a novel Transformer-based diffusion model
that uses LCU as input, aiming for joint training across various generation and
editing tasks. Furthermore, we propose an efficient data collection approach to
address the issue of the absence of available training data. It involves
acquiring pairwise images with synthesis-based or clustering-based pipelines
and supplying these pairs with accurate textual instructions by leveraging a
fine-tuned multi-modal large language model. To comprehensively evaluate the
performance of our model, we establish a benchmark of manually annotated pairs
data across a variety of visual generation tasks. The extensive experimental
results demonstrate the superiority of our model in visual generation fields.
Thanks to the all-in-one capabilities of our model, we can easily build a
multi-modal chat system that responds to any interactive request for image
creation using a single model to serve as the backend, avoiding the cumbersome
pipeline typically employed in visual agents. Code and models will be available
on the project page: https://ali-vilab.github.io/ace-page/.
ACE, a unified Transformer-based diffusion model, achieves comparable performance to expert models across various visual generation tasks by using a Long-context Condition Unit and addressing data scarcity with a novel data collection method.
diffusion models Transformer-based diffusion model Long-context Condition Unit multi-modal conditions visual generation multi-modal chat system