SNOOPI: Supercharged One-step Diffusion Distillation with Proper
  Guidance
Recent approaches have yielded promising results in distilling multi-step
text-to-image diffusion models into one-step ones. The state-of-the-art
efficient distillation technique, i.e., SwiftBrushv2 (SBv2), even surpasses the
teacher model's performance with limited resources. However, our study reveals
its instability when handling different diffusion model backbones due to using
a fixed guidance scale within the Variational Score Distillation (VSD) loss.
Another weakness of the existing one-step diffusion models is the missing
support for negative prompt guidance, which is crucial in practical image
generation. This paper presents SNOOPI, a novel framework designed to address
these limitations by enhancing the guidance in one-step diffusion models during
both training and inference. First, we effectively enhance training stability
through Proper Guidance-SwiftBrush (PG-SB), which employs a random-scale
classifier-free guidance approach. By varying the guidance scale of both
teacher models, we broaden their output distributions, resulting in a more
robust VSD loss that enables SB to perform effectively across diverse backbones
while maintaining competitive performance. Second, we propose a training-free
method called Negative-Away Steer Attention (NASA), which integrates negative
prompts into one-step diffusion models via cross-attention to suppress
undesired elements in generated images. Our experimental results show that our
proposed methods significantly improve baseline models across various metrics.
Remarkably, we achieve an HPSv2 score of 31.08, setting a new state-of-the-art
benchmark for one-step diffusion models.
674fd4fe696ab96e57021fbf
Viet Nguyen
claimed_verified
2024-12-04T09:44:28.182Z
643684cdc81d1646aa6b93a5
https://cdn-avatars.huggingface.co/v1/production/uploads/643684cdc81d1646aa6b93a5/iyLQfKMBCJJjfOJB2zvfp.png
Viet Nguyen
user
viettmab
674fd4fe696ab96e57021fc0
Anh Aengus Nguyen
claimed_verified
2024-12-04T09:44:30.839Z
6397329eb3d9042831c885af
https://cdn-avatars.huggingface.co/v1/production/uploads/6397329eb3d9042831c885af/sEIXpwBDv6dl5iUdsjA1I.jpeg
Anh Nguyen (Aengus)
user
aengusng
674fd4fe696ab96e57021fc1
Trung Dao
claimed_verified
2024-12-05T09:18:50.613Z
64d265019badd06a0587c848
https://cdn-avatars.huggingface.co/v1/production/uploads/64d265019badd06a0587c848/qrOtJ7gxUCzgVwduFDktf.jpeg
Dao Tuan Trung
user
termanteus
674fd4fe696ab96e57021fc2
Khoi Nguyen
674fd4fe696ab96e57021fc3
Cuong Pham
674fd4fe696ab96e57021fc4
Toan Tran
674fd4fe696ab96e57021fc5
Anh Tran
claimed_verified
2024-12-05T09:18:48.756Z
63e9fc6174f940d171e4c5e9
/avatars/0c34fdaecd7bd8aaffb3d68e83c0eeaf.svg
Anh Tran
user
anhttran1111
SNOOPI: Supercharged One-step Diffusion Distillation with Proper
  Guidance
Recent approaches have yielded promising results in distilling multi-step
text-to-image diffusion models into one-step ones. The state-of-the-art
efficient distillation technique, i.e., SwiftBrushv2 (SBv2), even surpasses the
teacher model's performance with limited resources. However, our study reveals
its instability when handling different diffusion model backbones due to using
a fixed guidance scale within the Variational Score Distillation (VSD) loss.
Another weakness of the existing one-step diffusion models is the missing
support for negative prompt guidance, which is crucial in practical image
generation. This paper presents SNOOPI, a novel framework designed to address
these limitations by enhancing the guidance in one-step diffusion models during
both training and inference. First, we effectively enhance training stability
through Proper Guidance-SwiftBrush (PG-SB), which employs a random-scale
classifier-free guidance approach. By varying the guidance scale of both
teacher models, we broaden their output distributions, resulting in a more
robust VSD loss that enables SB to perform effectively across diverse backbones
while maintaining competitive performance. Second, we propose a training-free
method called Negative-Away Steer Attention (NASA), which integrates negative
prompts into one-step diffusion models via cross-attention to suppress
undesired elements in generated images. Our experimental results show that our
proposed methods significantly improve baseline models across various metrics.
Remarkably, we achieve an HPSv2 score of 31.08, setting a new state-of-the-art
benchmark for one-step diffusion models.
SNOOPI framework improves one-step text-to-image diffusion models with Proper Guidance-SwiftBrush and Negative-Away Steer Attention, enhancing both training stability and negative prompt support.
text-to-image diffusion models SwiftBrushv2 Variational Score Distillation Proper Guidance-SwiftBrush Negative-Away Steer Attention cross-attention HPSv2