FreeControl: Training-Free Spatial Control of Any Text-to-Image
  Diffusion Model with Any Condition
Recent approaches such as ControlNet offer users fine-grained spatial control
over text-to-image (T2I) diffusion models. However, auxiliary modules have to
be trained for each type of spatial condition, model architecture, and
checkpoint, putting them at odds with the diverse intents and preferences a
human designer would like to convey to the AI models during the content
creation process. In this work, we present FreeControl, a training-free
approach for controllable T2I generation that supports multiple conditions,
architectures, and checkpoints simultaneously. FreeControl designs structure
guidance to facilitate the structure alignment with a guidance image, and
appearance guidance to enable the appearance sharing between images generated
using the same seed. Extensive qualitative and quantitative experiments
demonstrate the superior performance of FreeControl across a variety of
pre-trained T2I models. In particular, FreeControl facilitates convenient
training-free control over many different architectures and checkpoints, allows
the challenging input conditions on which most of the existing training-free
methods fail, and achieves competitive synthesis quality with training-based
approaches.
65794194c17736d75f8391aa
Sicheng Mo
claimed_verified
2023-12-13T08:32:29.083Z
637c94d3f219c71f93eda9ad
/avatars/6dae0c30755196ccc0a5a06b3981c47f.svg
Sicheng Mo
user
Sichengmo
65794194c17736d75f8391ab
Fangzhou Mu
admin_assigned
2023-12-13T08:58:39.371Z
65484f95c6dadd513f51f495
/avatars/cf7ea78cac16500df451556a071101e2.svg
Fangzhou Mu
user
fmu2
65794194c17736d75f8391ac
Kuan Heng Lin
claimed_verified
2023-12-13T08:32:34.357Z
638830946e484fea6da1783f
https://cdn-avatars.huggingface.co/v1/production/uploads/638830946e484fea6da1783f/-0KSxFtp9d_TVUo9HUKrK.png
Kuan Heng (Jordan) Lin
user
jordanlin
65794194c17736d75f8391ad
Yanli Liu
65794194c17736d75f8391ae
Bochen Guan
65794194c17736d75f8391af
Yin Li
claimed_verified
2023-12-14T10:34:43.710Z
6310311358d83e8f64dbc3f0
/avatars/85c0bbfddb44871cbefaa8dd17e3a246.svg
Yin Li
user
happyharrycn
65794194c17736d75f8391b0
Bolei Zhou
admin_assigned
2023-12-13T08:59:28.229Z
64aedf11c693a19547999a59
/avatars/83a54b7c217168c91cea15dd000f4af2.svg
Bolei Zhou
user
zhoubolei
FreeControl: Training-Free Spatial Control of Any Text-to-Image
  Diffusion Model with Any Condition
Recent approaches such as ControlNet offer users fine-grained spatial control
over text-to-image (T2I) diffusion models. However, auxiliary modules have to
be trained for each type of spatial condition, model architecture, and
checkpoint, putting them at odds with the diverse intents and preferences a
human designer would like to convey to the AI models during the content
creation process. In this work, we present FreeControl, a training-free
approach for controllable T2I generation that supports multiple conditions,
architectures, and checkpoints simultaneously. FreeControl designs structure
guidance to facilitate the structure alignment with a guidance image, and
appearance guidance to enable the appearance sharing between images generated
using the same seed. Extensive qualitative and quantitative experiments
demonstrate the superior performance of FreeControl across a variety of
pre-trained T2I models. In particular, FreeControl facilitates convenient
training-free control over many different architectures and checkpoints, allows
the challenging input conditions on which most of the existing training-free
methods fail, and achieves competitive synthesis quality with training-based
approaches.
FreeControl provides a training-free method to control text-to-image diffusion models with multiple conditions, architectures, and checkpoints, demonstrating competitive performance to training-based approaches.
ControlNet text-to-image (T2I) diffusion models free control structure guidance appearance guidance