Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion
  Models
Beyond high-fidelity image synthesis, diffusion models have recently
exhibited promising results in dense visual perception tasks. However, most
existing work treats diffusion models as a standalone component for perception
tasks, employing them either solely for off-the-shelf data augmentation or as
mere feature extractors. In contrast to these isolated and thus sub-optimal
efforts, we introduce a unified, versatile, diffusion-based framework,
Diff-2-in-1, that can simultaneously handle both multi-modal data generation
and dense visual perception, through a unique exploitation of the
diffusion-denoising process. Within this framework, we further enhance
discriminative visual perception via multi-modal generation, by utilizing the
denoising network to create multi-modal data that mirror the distribution of
the original training set. Importantly, Diff-2-in-1 optimizes the utilization
of the created diverse and faithful data by leveraging a novel self-improving
learning mechanism. Comprehensive experimental evaluations validate the
effectiveness of our framework, showcasing consistent performance improvements
across various discriminative backbones and high-quality multi-modal data
generation characterized by both realism and usefulness.
672e3a87b0d4d3107840222e
Shuhong Zheng
claimed_verified
2024-12-19T18:05:07.095Z
64361417a4bd75c62cc1e534
/avatars/b9b43985a0190ccf418c6853f9c3eb16.svg
Shuhong Zheng
user
ShuhongZheng
672e3a87b0d4d3107840222f
Zhipeng Bao
672e3a87b0d4d31078402230
Ruoyu Zhao
672e3a87b0d4d31078402231
Martial Hebert
672e3a87b0d4d31078402232
Yu-Xiong Wang
Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion
  Models
Beyond high-fidelity image synthesis, diffusion models have recently
exhibited promising results in dense visual perception tasks. However, most
existing work treats diffusion models as a standalone component for perception
tasks, employing them either solely for off-the-shelf data augmentation or as
mere feature extractors. In contrast to these isolated and thus sub-optimal
efforts, we introduce a unified, versatile, diffusion-based framework,
Diff-2-in-1, that can simultaneously handle both multi-modal data generation
and dense visual perception, through a unique exploitation of the
diffusion-denoising process. Within this framework, we further enhance
discriminative visual perception via multi-modal generation, by utilizing the
denoising network to create multi-modal data that mirror the distribution of
the original training set. Importantly, Diff-2-in-1 optimizes the utilization
of the created diverse and faithful data by leveraging a novel self-improving
learning mechanism. Comprehensive experimental evaluations validate the
effectiveness of our framework, showcasing consistent performance improvements
across various discriminative backbones and high-quality multi-modal data
generation characterized by both realism and usefulness.
Diff-2-in-1 is a unified diffusion-based framework that integrates multi-modal data generation and dense visual perception, improving performance through a self-improving mechanism.
diffusion models multi-modal data generation dense visual perception diffusion-denoising process denoising network self-improving learning mechanism discriminative backbones