ConsisLoRA: Enhancing Content and Style Consistency for LoRA-based Style
  Transfer
Style transfer involves transferring the style from a reference image to the
content of a target image. Recent advancements in LoRA-based (Low-Rank
Adaptation) methods have shown promise in effectively capturing the style of a
single image. However, these approaches still face significant challenges such
as content inconsistency, style misalignment, and content leakage. In this
paper, we comprehensively analyze the limitations of the standard diffusion
parameterization, which learns to predict noise, in the context of style
transfer. To address these issues, we introduce ConsisLoRA, a LoRA-based method
that enhances both content and style consistency by optimizing the LoRA weights
to predict the original image rather than noise. We also propose a two-step
training strategy that decouples the learning of content and style from the
reference image. To effectively capture both the global structure and local
details of the content image, we introduce a stepwise loss transition strategy.
Additionally, we present an inference guidance method that enables continuous
control over content and style strengths during inference. Through both
qualitative and quantitative evaluations, our method demonstrates significant
improvements in content and style consistency while effectively reducing
content leakage.
67d3d770336d57afb22fbe46
Bolin Chen
claimed_verified
2025-03-14T08:55:56.615Z
66e508b7ad382452bb326cbd
/avatars/e541e312c4fe9919778d266339d38206.svg
bolin
user
chenblin26
67d3d770336d57afb22fbe47
Baoquan Zhao
67d3d770336d57afb22fbe48
Haoran Xie
67d3d770336d57afb22fbe49
Yi Cai
67d3d770336d57afb22fbe4a
Qing Li
67d3d770336d57afb22fbe4b
Xudong Mao
ConsisLoRA: Enhancing Content and Style Consistency for LoRA-based Style
  Transfer
Style transfer involves transferring the style from a reference image to the
content of a target image. Recent advancements in LoRA-based (Low-Rank
Adaptation) methods have shown promise in effectively capturing the style of a
single image. However, these approaches still face significant challenges such
as content inconsistency, style misalignment, and content leakage. In this
paper, we comprehensively analyze the limitations of the standard diffusion
parameterization, which learns to predict noise, in the context of style
transfer. To address these issues, we introduce ConsisLoRA, a LoRA-based method
that enhances both content and style consistency by optimizing the LoRA weights
to predict the original image rather than noise. We also propose a two-step
training strategy that decouples the learning of content and style from the
reference image. To effectively capture both the global structure and local
details of the content image, we introduce a stepwise loss transition strategy.
Additionally, we present an inference guidance method that enables continuous
control over content and style strengths during inference. Through both
qualitative and quantitative evaluations, our method demonstrates significant
improvements in content and style consistency while effectively reducing
content leakage.
ConsisLoRA, a LoRA-based method, improves style transfer by predicting original images and separating content from style learning, with enhanced consistency and reduced leakage.
LoRA ConsisLoRA diffusion parameterization content consistency style consistency style misalignment content leakage two-step training stepwise loss transition inference guidance