Condition-Aware Neural Network for Controlled Image Generation
We present Condition-Aware Neural Network (CAN), a new method for adding
control to image generative models. In parallel to prior conditional control
methods, CAN controls the image generation process by dynamically manipulating
the weight of the neural network. This is achieved by introducing a
condition-aware weight generation module that generates conditional weight for
convolution/linear layers based on the input condition. We test CAN on
class-conditional image generation on ImageNet and text-to-image generation on
COCO. CAN consistently delivers significant improvements for diffusion
transformer models, including DiT and UViT. In particular, CAN combined with
EfficientViT (CaT) achieves 2.78 FID on ImageNet 512x512, surpassing DiT-XL/2
while requiring 52x fewer MACs per sampling step.
660b9407b29280b06b39ca8e
Han Cai
claimed_verified
2024-10-28T11:29:08.907Z
650e2b14c945dfc9386a7e28
/avatars/0a9be20aa53f4c52a2d1a8b02d4093ea.svg
Han Cai
user
han-cai
660b9407b29280b06b39ca8f
Muyang Li
claimed_verified
2024-11-15T09:48:01.510Z
63129589bbaa385279d1826e
https://cdn-avatars.huggingface.co/v1/production/uploads/63129589bbaa385279d1826e/0AUs3q4ngRZ-wXuY1jP9G.jpeg
Muyang Li
user
Lmxyy
660b9407b29280b06b39ca90
Zhuoyang Zhang
660b9407b29280b06b39ca91
Qinsheng Zhang
660b9407b29280b06b39ca92
Ming-Yu Liu
660b9407b29280b06b39ca93
Song Han
Condition-Aware Neural Network for Controlled Image Generation
We present Condition-Aware Neural Network (CAN), a new method for adding
control to image generative models. In parallel to prior conditional control
methods, CAN controls the image generation process by dynamically manipulating
the weight of the neural network. This is achieved by introducing a
condition-aware weight generation module that generates conditional weight for
convolution/linear layers based on the input condition. We test CAN on
class-conditional image generation on ImageNet and text-to-image generation on
COCO. CAN consistently delivers significant improvements for diffusion
transformer models, including DiT and UViT. In particular, CAN combined with
EfficientViT (CaT) achieves 2.78 FID on ImageNet 512x512, surpassing DiT-XL/2
while requiring 52x fewer MACs per sampling step.
A Condition-Aware Neural Network (CAN) method for image generation dynamically adjusts neural network weights based on input conditions, demonstrating significant improvements for diffusion transformer models and surpassing existing models in terms of FID score and computational efficiency.
Condition-Aware Neural Network (CAN) condition-aware weight generation module diffusion transformer models DiT UViT EfficientViT (CaT) FID MACs