Unlocking Pre-trained Image Backbones for Semantic Image Synthesis
Semantic image synthesis, i.e., generating images from user-provided semantic
label maps, is an important conditional image generation task as it allows to
control both the content as well as the spatial layout of generated images.
Although diffusion models have pushed the state of the art in generative image
modeling, the iterative nature of their inference process makes them
computationally demanding. Other approaches such as GANs are more efficient as
they only need a single feed-forward pass for generation, but the image quality
tends to suffer on large and diverse datasets. In this work, we propose a new
class of GAN discriminators for semantic image synthesis that generates highly
realistic images by exploiting feature backbone networks pre-trained for tasks
such as image classification. We also introduce a new generator architecture
with better context modeling and using cross-attention to inject noise into
latent variables, leading to more diverse generated images. Our model, which we
dub DP-SIMS, achieves state-of-the-art results in terms of image quality and
consistency with the input label maps on ADE-20K, COCO-Stuff, and Cityscapes,
surpassing recent diffusion models while requiring two orders of magnitude less
compute for inference.
658510d7c7918aa044940648
Tariq Berrada
claimed_verified
2023-12-22T12:33:50.167Z
65858134e07563d421117279
/avatars/08fb3f98cd96aa1828f5b9929b30cbb7.svg
Tariq Berrada
user
TariqBerrada
658510d7c7918aa044940649
Jakob Verbeek
extracted_confirmed
2024-10-16T15:59:29.103Z
670fe1c99e1efdfc2ea39665
/avatars/78d777c0bd6c490a1e5ba7ad7d038335.svg
Jakob Verbeek
user
jjverbeek
658510d7c7918aa04494064a
Camille Couprie
658510d7c7918aa04494064b
Karteek Alahari
Unlocking Pre-trained Image Backbones for Semantic Image Synthesis
Semantic image synthesis, i.e., generating images from user-provided semantic
label maps, is an important conditional image generation task as it allows to
control both the content as well as the spatial layout of generated images.
Although diffusion models have pushed the state of the art in generative image
modeling, the iterative nature of their inference process makes them
computationally demanding. Other approaches such as GANs are more efficient as
they only need a single feed-forward pass for generation, but the image quality
tends to suffer on large and diverse datasets. In this work, we propose a new
class of GAN discriminators for semantic image synthesis that generates highly
realistic images by exploiting feature backbone networks pre-trained for tasks
such as image classification. We also introduce a new generator architecture
with better context modeling and using cross-attention to inject noise into
latent variables, leading to more diverse generated images. Our model, which we
dub DP-SIMS, achieves state-of-the-art results in terms of image quality and
consistency with the input label maps on ADE-20K, COCO-Stuff, and Cityscapes,
surpassing recent diffusion models while requiring two orders of magnitude less
compute for inference.
A new GAN-based model, DP-SIMS, achieves high-quality and consistent semantic image synthesis with lower computational cost by using pre-trained feature backbones and cross-attention in the generator.
diffusion models GANs feature backbone networks cross-attention DP-SIMS semantic image synthesis ADE-20K COCO-Stuff Cityscapes