Noise Consistency Training: A Native Approach for One-Step Generator in
  Learning Additional Controls
The pursuit of efficient and controllable high-quality content generation
remains a central challenge in artificial intelligence-generated content
(AIGC). While one-step generators, enabled by diffusion distillation
techniques, offer excellent generation quality and computational efficiency,
adapting them to new control conditions--such as structural constraints,
semantic guidelines, or external inputs--poses a significant challenge.
Conventional approaches often necessitate computationally expensive
modifications to the base model and subsequent diffusion distillation. This
paper introduces Noise Consistency Training (NCT), a novel and lightweight
approach to directly integrate new control signals into pre-trained one-step
generators without requiring access to original training images or retraining
the base diffusion model. NCT operates by introducing an adapter module and
employs a noise consistency loss in the noise space of the generator. This loss
aligns the adapted model's generation behavior across noises that are
conditionally dependent to varying degrees, implicitly guiding it to adhere to
the new control. Theoretically, this training objective can be understood as
minimizing the distributional distance between the adapted generator and the
conditional distribution induced by the new conditions. NCT is modular,
data-efficient, and easily deployable, relying only on the pre-trained one-step
generator and a control signal model. Extensive experiments demonstrate that
NCT achieves state-of-the-art controllable generation in a single forward pass,
surpassing existing multi-step and distillation-based methods in both
generation quality and computational efficiency. Code is available at
https://github.com/Luo-Yihong/NCT
686209869e7509383d29ab92
Yihong Luo
686209869e7509383d29ab93
Shuchen Xue
686209869e7509383d29ab94
Tianyang Hu
686209869e7509383d29ab95
Jing Tang
Noise Consistency Training: A Native Approach for One-Step Generator in
  Learning Additional Controls
The pursuit of efficient and controllable high-quality content generation
remains a central challenge in artificial intelligence-generated content
(AIGC). While one-step generators, enabled by diffusion distillation
techniques, offer excellent generation quality and computational efficiency,
adapting them to new control conditions--such as structural constraints,
semantic guidelines, or external inputs--poses a significant challenge.
Conventional approaches often necessitate computationally expensive
modifications to the base model and subsequent diffusion distillation. This
paper introduces Noise Consistency Training (NCT), a novel and lightweight
approach to directly integrate new control signals into pre-trained one-step
generators without requiring access to original training images or retraining
the base diffusion model. NCT operates by introducing an adapter module and
employs a noise consistency loss in the noise space of the generator. This loss
aligns the adapted model's generation behavior across noises that are
conditionally dependent to varying degrees, implicitly guiding it to adhere to
the new control. Theoretically, this training objective can be understood as
minimizing the distributional distance between the adapted generator and the
conditional distribution induced by the new conditions. NCT is modular,
data-efficient, and easily deployable, relying only on the pre-trained one-step
generator and a control signal model. Extensive experiments demonstrate that
NCT achieves state-of-the-art controllable generation in a single forward pass,
surpassing existing multi-step and distillation-based methods in both
generation quality and computational efficiency. Code is available at
https://github.com/Luo-Yihong/NCT
A novel Noise Consistency Training approach integrates new control signals into pre-trained one-step generators efficiently without retraining, outperforming existing methods in quality and computational efficiency.
diffusion distillation Noise Consistency Training NCT one-step generators adapter module noise consistency loss noise space conditional distribution generative modeling data-efficient computational efficiency