DICEPTION: A Generalist Diffusion Model for Visual Perceptual Tasks
Our primary goal here is to create a good, generalist perception model that
can tackle multiple tasks, within limits on computational resources and
training data. To achieve this, we resort to text-to-image diffusion models
pre-trained on billions of images. Our exhaustive evaluation metrics
demonstrate that DICEPTION effectively tackles multiple perception tasks,
achieving performance on par with state-of-the-art models. We achieve results
on par with SAM-vit-h using only 0.06% of their data (e.g., 600K vs. 1B
pixel-level annotated images). Inspired by Wang et al., DICEPTION formulates
the outputs of various perception tasks using color encoding; and we show that
the strategy of assigning random colors to different instances is highly
effective in both entity segmentation and semantic segmentation. Unifying
various perception tasks as conditional image generation enables us to fully
leverage pre-trained text-to-image models. Thus, DICEPTION can be efficiently
trained at a cost of orders of magnitude lower, compared to conventional models
that were trained from scratch. When adapting our model to other tasks, it only
requires fine-tuning on as few as 50 images and 1% of its parameters. DICEPTION
provides valuable insights and a more promising solution for visual generalist
models.
67bd3285ac4a596a43b53205
Canyu Zhao
claimed_verified
2025-02-25T09:40:20.829Z
646efd223dd912a539e0bd46
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/EOFAv5xvOgJOzuDgh4nSb.png
Canyu Zhao
user
Canyu
67bd3285ac4a596a43b53206
Mingyu Liu
claimed_verified
2025-02-26T08:38:01.503Z
652e25d2e647b0ee0a024f26
/avatars/b5c65cf6c8d0ddc9b8ef0226e0295d56.svg
Mingyu Liu
user
MingyuLiu
67bd3285ac4a596a43b53207
Huanyi Zheng
claimed_verified
2025-02-25T09:40:18.731Z
64d60375d7e30889c65e8cf4
/avatars/640f7c570fc45194557ce7931bdfe87f.svg
Huanyi Zheng
user
zhyya
67bd3285ac4a596a43b53208
Muzhi Zhu
claimed_verified
2025-02-25T09:40:11.968Z
632179745fc60c44fd91fc33
/avatars/37d4fefbcc19f091dccffefec9706de2.svg
zhumuzhi
user
Z-MU-Z
67bd3285ac4a596a43b53209
Zhiyue Zhao
67bd3285ac4a596a43b5320a
Hao Chen
67bd3285ac4a596a43b5320b
Tong He
claimed_verified
2025-07-17T13:57:15.198Z
6747ede3a9c72aebe1322382
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/inILqQ05sESbYLdsEldJ_.png
Tong He
user
tonghe90
67bd3285ac4a596a43b5320c
Chunhua Shen
DICEPTION: A Generalist Diffusion Model for Visual Perceptual Tasks
Our primary goal here is to create a good, generalist perception model that
can tackle multiple tasks, within limits on computational resources and
training data. To achieve this, we resort to text-to-image diffusion models
pre-trained on billions of images. Our exhaustive evaluation metrics
demonstrate that DICEPTION effectively tackles multiple perception tasks,
achieving performance on par with state-of-the-art models. We achieve results
on par with SAM-vit-h using only 0.06% of their data (e.g., 600K vs. 1B
pixel-level annotated images). Inspired by Wang et al., DICEPTION formulates
the outputs of various perception tasks using color encoding; and we show that
the strategy of assigning random colors to different instances is highly
effective in both entity segmentation and semantic segmentation. Unifying
various perception tasks as conditional image generation enables us to fully
leverage pre-trained text-to-image models. Thus, DICEPTION can be efficiently
trained at a cost of orders of magnitude lower, compared to conventional models
that were trained from scratch. When adapting our model to other tasks, it only
requires fine-tuning on as few as 50 images and 1% of its parameters. DICEPTION
provides valuable insights and a more promising solution for visual generalist
models.
DICEPTION, a text-to-image diffusion model, achieves state-of-the-art performance in multiple perception tasks using minimal data and computational resources by leveraging color encoding and conditional image generation.
text-to-image diffusion models perception tasks color encoding conditional image generation