StreamMultiDiffusion: Real-Time Interactive Generation with Region-Based
  Semantic Control
The enormous success of diffusion models in text-to-image synthesis has made
them promising candidates for the next generation of end-user applications for
image generation and editing. Previous works have focused on improving the
usability of diffusion models by reducing the inference time or increasing user
interactivity by allowing new, fine-grained controls such as region-based text
prompts. However, we empirically find that integrating both branches of works
is nontrivial, limiting the potential of diffusion models. To solve this
incompatibility, we present StreamMultiDiffusion, the first real-time
region-based text-to-image generation framework. By stabilizing fast inference
techniques and restructuring the model into a newly proposed multi-prompt
stream batch architecture, we achieve times 10 faster panorama generation
than existing solutions, and the generation speed of 1.57 FPS in region-based
text-to-image synthesis on a single RTX 2080 Ti GPU. Our solution opens up a
new paradigm for interactive image generation named semantic palette, where
high-quality images are generated in real-time from given multiple hand-drawn
regions, encoding prescribed semantic meanings (e.g., eagle, girl). Our code
and demo application are available at
https://github.com/ironjr/StreamMultiDiffusion.
65f3caf7c4a561cc64d832d0
Jaerin Lee
admin_assigned
2024-03-15T08:25:12.689Z
643bcf4786ab6dbe34ecae59
/avatars/4fab7726c0eb5f26b2e741d1d6bf2edb.svg
Jaerin Lee
user
ironjr
65f3caf7c4a561cc64d832d1
Daniel Sungho Jung
claimed_verified
2024-03-15T08:03:06.001Z
65601c6ee23401f82005e361
/avatars/e9fc24bd8c5afd8b07a2f42765d44a7d.svg
Daniel Sungho Jung
user
dqj5182
65f3caf7c4a561cc64d832d2
Kanggeon Lee
65f3caf7c4a561cc64d832d3
Kyoung Mu Lee
admin_assigned
2024-03-15T08:25:22.867Z
656056b21392aa3beb5de0bd
/avatars/07f25b750ef308d65f2e6c82506e7816.svg
Kyoung Mu  Lee 
user
kyoungmu
StreamMultiDiffusion: Real-Time Interactive Generation with Region-Based
  Semantic Control
The enormous success of diffusion models in text-to-image synthesis has made
them promising candidates for the next generation of end-user applications for
image generation and editing. Previous works have focused on improving the
usability of diffusion models by reducing the inference time or increasing user
interactivity by allowing new, fine-grained controls such as region-based text
prompts. However, we empirically find that integrating both branches of works
is nontrivial, limiting the potential of diffusion models. To solve this
incompatibility, we present StreamMultiDiffusion, the first real-time
region-based text-to-image generation framework. By stabilizing fast inference
techniques and restructuring the model into a newly proposed multi-prompt
stream batch architecture, we achieve times 10 faster panorama generation
than existing solutions, and the generation speed of 1.57 FPS in region-based
text-to-image synthesis on a single RTX 2080 Ti GPU. Our solution opens up a
new paradigm for interactive image generation named semantic palette, where
high-quality images are generated in real-time from given multiple hand-drawn
regions, encoding prescribed semantic meanings (e.g., eagle, girl). Our code
and demo application are available at
https://github.com/ironjr/StreamMultiDiffusion.
StreamMultiDiffusion is a real-time region-based text-to-image generation framework that achieves fast inference speeds using a multi-prompt stream batch architecture, enabling interactive image generation.
diffusion models text-to-image synthesis inference time real-time region-based text-to-image multi-prompt stream batch architecture panorama generation semantic palette hand-drawn regions semantic meanings