ScaleDiff: Higher-Resolution Image Synthesis via Efficient and Model-Agnostic Diffusion
Sungho Koh, SeungJu Cha, Hyunwoo Oh, Kwanyoung Lee, Dong-Jin Kim
Text-to-image diffusion models often exhibit degraded performance when generating images beyond their training resolution.Recent training-free methods can mitigate this limitation, but they are primarily designed for U-Net-based models and often yield suboptimal results when applied to recent Diffusion Transformer models.While patch-based methods are architecture-agnostic, they incur substantial computational due to overlapping patches.In this paper, we propose ScaleDiff, a model-agnostic and highly efficient framework for extending the resolution of pretrained diffusion models without any additional training.A core component of our framework is Neighborhood Patch Attention (NPA), an efficient mechanism that reduces computational redundancy in self-attention layer with non-overlapping patches.We integrate NPA into an upsample-diffuse-denoise pipeline and introduce Latent Frequency Mixing (LFM) to better generate fine details.Furthermore, we apply Structure Guidance to enhance global structure during the denoising process.Experimental results demonstrate that ScaleDiff achieves state-of-the-art performance among training-free methods in terms of both image quality and inference speed on both U-Net and Diffusion Transformer architectures.