Preventing Shortcuts in Adapter Training via .. well .. Providing the Shortcuts
Anujraaj Goyal, Guocheng Qian, n n, Aarush Gupta, Himmy Tam, Daniil Ostashev, Ju Hu, Dhritiman Sagar, Sergey Tulyakov, Kfir Aberman, Kuan-Chieh Wang
Adapter-based training has emerged as a key mechanism for extending the capabilities of powerful foundation image generators, enabling personalized and stylized text-to-image synthesis.These adapters are typically trained to capture a specific target attribute, such as subject identity, using single-image reconstruction objectives. However, because the input image inevitably contains a mixture of visual factors, adapters are prone to entangling the target attribute with incidental ones such as pose, expression, or lighting. This spurious correlation problem limits generalization and undermines modularity: an identity adapter unwittingly encodes pose and expression, failing to adhere to prompts that require changing the subject's pose and expression. In this work, we uncover a simple yet effective solution: provide the very shortcuts we wish to eliminate. In *Shortcut-Rerouted Adapter Training*, confounding factors are routed through auxiliary modules, such as ControlNet or LoRA, during training, eliminating the incentive for the adapter to internalize them. Applied to tasks like facial and full-body identity injection, our approach improves generation quality, diversity, and prompt adherence. These results point to a general design principle in the era of large models: when seeking disentangled representations, the most effective path may be to establish shortcuts for what should *not* be learned.