Universal Few-shot Spatial Control for Diffusion Models
Kiet Nguyen, Chanhyuk Lee, Donggyun Kim, Dong Hoon Lee, Seunghoon Hong
Spatial conditioning in pretrained text-to-image diffusion models has significantly improved fine-grained control over the structure of generated images. However, existing control adapters exhibit limited adaptability and incur high training costs when encountering novel spatial control conditions that differ substantially from the training tasks.To address this limitation, we propose Universal Few-Shot Control (UFC), a versatile few-shot control adapter capable of generalizing to novel spatial conditions.Given a few image-condition pairs of an unseen task and a query condition, UFC leverage the analogy between query and support conditions to construct task-specific control feature, instantiated by a matching mechanism and an update on a small set of task-specific parameters. Experiments on six spatial control tasks show that UFC, finetuned with only 30 annotated examples, achieves fine-grained control consistent with the spatial conditions.Notably, when finetuned with 0.1% of the full training data, UFC can even be on par with the fully supervised baseline, i.e., Uni-ControlNet, on the Normal, Depth, and Canny tasks.These results highlight UFC's effectiveness in controlling diffusion models when faced with novel spatial conditions.