T-LoRA: Single Image Diffusion Model Customization Without Overfitting
While diffusion model fine-tuning offers a powerful approach for customizing
pre-trained models to generate specific objects, it frequently suffers from
overfitting when training samples are limited, compromising both generalization
capability and output diversity. This paper tackles the challenging yet most
impactful task of adapting a diffusion model using just a single concept image,
as single-image customization holds the greatest practical potential. We
introduce T-LoRA, a Timestep-Dependent Low-Rank Adaptation framework
specifically designed for diffusion model personalization. In our work we show
that higher diffusion timesteps are more prone to overfitting than lower ones,
necessitating a timestep-sensitive fine-tuning strategy. T-LoRA incorporates
two key innovations: (1) a dynamic fine-tuning strategy that adjusts
rank-constrained updates based on diffusion timesteps, and (2) a weight
parametrization technique that ensures independence between adapter components
through orthogonal initialization. Extensive experiments show that T-LoRA and
its individual components outperform standard LoRA and other diffusion model
personalization techniques. They achieve a superior balance between concept
fidelity and text alignment, highlighting the potential of T-LoRA in
data-limited and resource-constrained scenarios. Code is available at
https://github.com/ControlGenAI/T-LoRA.
6870b8b5c8391850d60978e0
Vera Soboleva
claimed_verified
2025-07-11T16:31:53.439Z
6385d420c12615765caa00b2
/avatars/c125b2d61423908d453231b046fb6a28.svg
Vera Soboleva
user
verasobol
6870b8b5c8391850d60978e1
Aibek Alanov
claimed_verified
2025-07-11T07:59:56.813Z
66680c6451545a8b46c6fd21
/avatars/03707f5ea4e2aa8dc825a9782b00ed85.svg
Aibek Alanov
user
ai-alanov
6870b8b5c8391850d60978e2
Andrey Kuznetsov
claimed_verified
2025-07-11T13:12:36.562Z
643984dceb7c5616ef3f5d54
https://cdn-avatars.huggingface.co/v1/production/uploads/643984dceb7c5616ef3f5d54/10JRkblrRIEVci6UJwvPz.jpeg
Andrey Kuznetsov
user
kuznetsoffandrey
6870b8b5c8391850d60978e3
Konstantin Sobolev
claimed_verified
2025-07-11T16:31:55.163Z
6388ba34ec1f539adc092b56
/avatars/1d87a657dd81e6ca025cc020f3205525.svg
Konstantin Sobolev
user
k-sobolev
T-LoRA: Single Image Diffusion Model Customization Without Overfitting
While diffusion model fine-tuning offers a powerful approach for customizing
pre-trained models to generate specific objects, it frequently suffers from
overfitting when training samples are limited, compromising both generalization
capability and output diversity. This paper tackles the challenging yet most
impactful task of adapting a diffusion model using just a single concept image,
as single-image customization holds the greatest practical potential. We
introduce T-LoRA, a Timestep-Dependent Low-Rank Adaptation framework
specifically designed for diffusion model personalization. In our work we show
that higher diffusion timesteps are more prone to overfitting than lower ones,
necessitating a timestep-sensitive fine-tuning strategy. T-LoRA incorporates
two key innovations: (1) a dynamic fine-tuning strategy that adjusts
rank-constrained updates based on diffusion timesteps, and (2) a weight
parametrization technique that ensures independence between adapter components
through orthogonal initialization. Extensive experiments show that T-LoRA and
its individual components outperform standard LoRA and other diffusion model
personalization techniques. They achieve a superior balance between concept
fidelity and text alignment, highlighting the potential of T-LoRA in
data-limited and resource-constrained scenarios. Code is available at
https://github.com/ControlGenAI/T-LoRA.
T-LoRA, a timestep-dependent low-rank adaptation framework, enhances diffusion model personalization with a dynamic fine-tuning strategy and orthogonal initialization, improving concept fidelity and text alignment in data-limited settings.
diffusion model fine-tuning overfitting single concept image T-LoRA timestep-dependent low-rank adaptation dynamic fine-tuning rank-constrained updates orthogonal initialization concept fidelity text alignment