SPF-Portrait: Towards Pure Portrait Customization with Semantic
  Pollution-Free Fine-tuning
Fine-tuning a pre-trained Text-to-Image (T2I) model on a tailored portrait
dataset is the mainstream method for text-driven customization of portrait
attributes. Due to Semantic Pollution during fine-tuning, existing methods
struggle to maintain the original model's behavior and achieve incremental
learning while customizing target attributes. To address this issue, we propose
SPF-Portrait, a pioneering work to purely understand customized semantics while
eliminating semantic pollution in text-driven portrait customization. In our
SPF-Portrait, we propose a dual-path pipeline that introduces the original
model as a reference for the conventional fine-tuning path. Through contrastive
learning, we ensure adaptation to target attributes and purposefully align
other unrelated attributes with the original portrait. We introduce a novel
Semantic-Aware Fine Control Map, which represents the precise response regions
of the target semantics, to spatially guide the alignment process between the
contrastive paths. This alignment process not only effectively preserves the
performance of the original model but also avoids over-alignment. Furthermore,
we propose a novel response enhancement mechanism to reinforce the performance
of target attributes, while mitigating representation discrepancy inherent in
direct cross-modal supervision. Extensive experiments demonstrate that
SPF-Portrait achieves state-of-the-art performance. Project webpage:
https://spf-portrait.github.io/SPF-Portrait/
67f3c4f40db583bb63e03152
Xiaole Xian
claimed_verified
2025-04-09T16:22:43.251Z
651e27b6f6663789c8e9c5f5
/avatars/539498d74844aa08741d3fd63b0aa96f.svg
Xian Xiaole
user
connorxian
67f3c4f40db583bb63e03153
Zhichao Liao
claimed_verified
2025-04-08T06:52:45.276Z
66eb9d5d7f17088991ecbae2
https://cdn-avatars.huggingface.co/v1/production/uploads/66eb9d5d7f17088991ecbae2/khI2KZ4ZGJ891iOf-O0BF.jpeg
Zhichao Liao
user
ChaosLiao
67f3c4f40db583bb63e03154
Qingyu Li
67f3c4f40db583bb63e03155
Wenyu Qin
67f3c4f40db583bb63e03156
Pengfei Wan
claimed_verified
2025-05-04T10:05:44.304Z
662f93942510ef5735d7ad00
https://cdn-avatars.huggingface.co/v1/production/uploads/662f93942510ef5735d7ad00/ZIDIPm63sncIHFTT5b0uR.png
magicwpf
user
magicwpf
67f3c4f40db583bb63e03157
Weicheng Xie
67f3c4f40db583bb63e03158
Long Zeng
67f3c4f40db583bb63e03159
Linlin Shen
67f3c4f40db583bb63e0315a
Pingfa Feng
SPF-Portrait: Towards Pure Portrait Customization with Semantic
  Pollution-Free Fine-tuning
Fine-tuning a pre-trained Text-to-Image (T2I) model on a tailored portrait
dataset is the mainstream method for text-driven customization of portrait
attributes. Due to Semantic Pollution during fine-tuning, existing methods
struggle to maintain the original model's behavior and achieve incremental
learning while customizing target attributes. To address this issue, we propose
SPF-Portrait, a pioneering work to purely understand customized semantics while
eliminating semantic pollution in text-driven portrait customization. In our
SPF-Portrait, we propose a dual-path pipeline that introduces the original
model as a reference for the conventional fine-tuning path. Through contrastive
learning, we ensure adaptation to target attributes and purposefully align
other unrelated attributes with the original portrait. We introduce a novel
Semantic-Aware Fine Control Map, which represents the precise response regions
of the target semantics, to spatially guide the alignment process between the
contrastive paths. This alignment process not only effectively preserves the
performance of the original model but also avoids over-alignment. Furthermore,
we propose a novel response enhancement mechanism to reinforce the performance
of target attributes, while mitigating representation discrepancy inherent in
direct cross-modal supervision. Extensive experiments demonstrate that
SPF-Portrait achieves state-of-the-art performance. Project webpage:
https://spf-portrait.github.io/SPF-Portrait/
SPF-Portrait addresses semantic pollution during fine-tuning by using a dual-path pipeline with contrastive learning and a Semantic-Aware Fine Control Map for accurate spatial alignment, enhancing target attributes while preserving the original model's performance.
fine-tuning pre-trained model Text-to-Image (T2I) Semantic Pollution dual-path pipeline contrastive learning Semantic-Aware Fine Control Map spatial alignment over-alignment response enhancement mechanism cross-modal supervision