Efficient Distillation of Classifier-Free Guidance using Adapters
While classifier-free guidance (CFG) is essential for conditional diffusion
models, it doubles the number of neural function evaluations (NFEs) per
inference step. To mitigate this inefficiency, we introduce adapter guidance
distillation (AGD), a novel approach that simulates CFG in a single forward
pass. AGD leverages lightweight adapters to approximate CFG, effectively
doubling the sampling speed while maintaining or even improving sample quality.
Unlike prior guidance distillation methods that tune the entire model, AGD
keeps the base model frozen and only trains minimal additional parameters
(sim2%) to significantly reduce the resource requirement of the distillation
phase. Additionally, this approach preserves the original model weights and
enables the adapters to be seamlessly combined with other checkpoints derived
from the same base model. We also address a key mismatch between training and
inference in existing guidance distillation methods by training on CFG-guided
trajectories instead of standard diffusion trajectories. Through extensive
experiments, we show that AGD achieves comparable or superior FID to CFG across
multiple architectures with only half the NFEs. Notably, our method enables the
distillation of large models (sim2.6B parameters) on a single consumer GPU
with 24 GB of VRAM, making it more accessible than previous approaches that
require multiple high-end GPUs. We will publicly release the implementation of
our method.
67d010ecdf1cf0ec0b961fd1
Cristian Perez Jensen
claimed_verified
2025-03-11T10:57:59.848Z
65e1863062dc883757896f98
/avatars/125c56f4c605a09485e0b17e76897bdf.svg
Cristian Perez Jensen
user
cristianpjensen
67d010ecdf1cf0ec0b961fd2
Seyedmorteza Sadat
claimed_verified
2025-03-11T10:37:28.600Z
63b4b02a103617b0a5b0ee2e
https://cdn-avatars.huggingface.co/v1/production/uploads/63b4b02a103617b0a5b0ee2e/LasBC-pCnPJ6XuCt6qqcp.jpeg
Seyedmorteza Sadat
user
msadat97
Efficient Distillation of Classifier-Free Guidance using Adapters
While classifier-free guidance (CFG) is essential for conditional diffusion
models, it doubles the number of neural function evaluations (NFEs) per
inference step. To mitigate this inefficiency, we introduce adapter guidance
distillation (AGD), a novel approach that simulates CFG in a single forward
pass. AGD leverages lightweight adapters to approximate CFG, effectively
doubling the sampling speed while maintaining or even improving sample quality.
Unlike prior guidance distillation methods that tune the entire model, AGD
keeps the base model frozen and only trains minimal additional parameters
(sim2%) to significantly reduce the resource requirement of the distillation
phase. Additionally, this approach preserves the original model weights and
enables the adapters to be seamlessly combined with other checkpoints derived
from the same base model. We also address a key mismatch between training and
inference in existing guidance distillation methods by training on CFG-guided
trajectories instead of standard diffusion trajectories. Through extensive
experiments, we show that AGD achieves comparable or superior FID to CFG across
multiple architectures with only half the NFEs. Notably, our method enables the
distillation of large models (sim2.6B parameters) on a single consumer GPU
with 24 GB of VRAM, making it more accessible than previous approaches that
require multiple high-end GPUs. We will publicly release the implementation of
our method.
Adapter guidance distillation reduces the number of function evaluations in conditional diffusion models, enhancing sampling speed and preserving model quality with minimal additional parameters.
classifier-free guidance conditional diffusion models neural function evaluations adapter guidance distillation lightweight adapters guidance distillation sampling speed model weights CFG-guided trajectories standard diffusion trajectories FID