One-dimensional Adapter to Rule Them All: Concepts, Diffusion Models and
  Erasing Applications
The prevalent use of commercial and open-source diffusion models (DMs) for
text-to-image generation prompts risk mitigation to prevent undesired
behaviors. Existing concept erasing methods in academia are all based on full
parameter or specification-based fine-tuning, from which we observe the
following issues: 1) Generation alternation towards erosion: Parameter drift
during target elimination causes alternations and potential deformations across
all generations, even eroding other concepts at varying degrees, which is more
evident with multi-concept erased; 2) Transfer inability & deployment
inefficiency: Previous model-specific erasure impedes the flexible combination
of concepts and the training-free transfer towards other models, resulting in
linear cost growth as the deployment scenarios increase. To achieve
non-invasive, precise, customizable, and transferable elimination, we ground
our erasing framework on one-dimensional adapters to erase multiple concepts
from most DMs at once across versatile erasing applications. The
concept-SemiPermeable structure is injected as a Membrane (SPM) into any DM to
learn targeted erasing, and meantime the alteration and erosion phenomenon is
effectively mitigated via a novel Latent Anchoring fine-tuning strategy. Once
obtained, SPMs can be flexibly combined and plug-and-play for other DMs without
specific re-tuning, enabling timely and efficient adaptation to diverse
scenarios. During generation, our Facilitated Transport mechanism dynamically
regulates the permeability of each SPM to respond to different input prompts,
further minimizing the impact on other concepts. Quantitative and qualitative
results across ~40 concepts, 7 DMs and 4 erasing applications have demonstrated
the superior erasing of SPM. Our code and pre-tuned SPMs will be available on
the project page https://lyumengyao.github.io/projects/spm.
658bb39370d10c8982e4f7fe
Mengyao Lyu
extracted_pending
2025-04-02T02:18:00.525Z
67ec9d4d077a9bc63e725915
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/FlNOSbACDzbKztI3_hXOP.png
Mengyao Lyu
user
mengyaolyu
658bb39370d10c8982e4f7ff
Yuhong Yang
658bb39370d10c8982e4f800
Haiwen Hong
658bb39370d10c8982e4f801
Hui Chen
658bb39370d10c8982e4f802
Xuan Jin
658bb39370d10c8982e4f803
Yuan He
658bb39370d10c8982e4f804
Hui Xue
658bb39370d10c8982e4f805
Jungong Han
658bb39370d10c8982e4f806
Guiguang Ding
One-dimensional Adapter to Rule Them All: Concepts, Diffusion Models and
  Erasing Applications
The prevalent use of commercial and open-source diffusion models (DMs) for
text-to-image generation prompts risk mitigation to prevent undesired
behaviors. Existing concept erasing methods in academia are all based on full
parameter or specification-based fine-tuning, from which we observe the
following issues: 1) Generation alternation towards erosion: Parameter drift
during target elimination causes alternations and potential deformations across
all generations, even eroding other concepts at varying degrees, which is more
evident with multi-concept erased; 2) Transfer inability & deployment
inefficiency: Previous model-specific erasure impedes the flexible combination
of concepts and the training-free transfer towards other models, resulting in
linear cost growth as the deployment scenarios increase. To achieve
non-invasive, precise, customizable, and transferable elimination, we ground
our erasing framework on one-dimensional adapters to erase multiple concepts
from most DMs at once across versatile erasing applications. The
concept-SemiPermeable structure is injected as a Membrane (SPM) into any DM to
learn targeted erasing, and meantime the alteration and erosion phenomenon is
effectively mitigated via a novel Latent Anchoring fine-tuning strategy. Once
obtained, SPMs can be flexibly combined and plug-and-play for other DMs without
specific re-tuning, enabling timely and efficient adaptation to diverse
scenarios. During generation, our Facilitated Transport mechanism dynamically
regulates the permeability of each SPM to respond to different input prompts,
further minimizing the impact on other concepts. Quantitative and qualitative
results across ~40 concepts, 7 DMs and 4 erasing applications have demonstrated
the superior erasing of SPM. Our code and pre-tuned SPMs will be available on
the project page https://lyumengyao.github.io/projects/spm.
A novel erasing framework using one-dimensional adapters and Latent Anchoring fine-tuning strategy mitigates undesired concept alternations and enhances transferability in diffusion models for text-to-image generation.
diffusion models concept erasing one-dimensional adapters Membrane (SPM) Latent Anchoring fine-tuning strategy Facilitated Transport mechanism