MONKEY: Masking ON KEY-Value Activation Adapter for Personalization
Personalizing diffusion models allows users to generate new images that
incorporate a given subject, allowing more control than a text prompt. These
models often suffer somewhat when they end up just recreating the subject
image, and ignoring the text prompt. We observe that one popular method for
personalization, the IP-Adapter automatically generates masks that we
definitively segment the subject from the background during inference. We
propose to use this automatically generated mask on a second pass to mask the
image tokens, thus restricting them to the subject, not the background,
allowing the text prompt to attend to the rest of the image. For text prompts
describing locations and places, this produces images that accurately depict
the subject while definitively matching the prompt. We compare our method to a
few other test time personalization methods, and find our method displays high
prompt and source image alignment.
68ed1788de1fee572713a55e
James Baker
claimed_verified
2025-10-13T16:08:30.851Z
637317efbd81fae2b3aa400c
/avatars/898fa184c2d2fb7479ed097b4ddee5e9.svg
James Baker
user
jlbaker361
MONKEY: Masking ON KEY-Value Activation Adapter for Personalization
Personalizing diffusion models allows users to generate new images that
incorporate a given subject, allowing more control than a text prompt. These
models often suffer somewhat when they end up just recreating the subject
image, and ignoring the text prompt. We observe that one popular method for
personalization, the IP-Adapter automatically generates masks that we
definitively segment the subject from the background during inference. We
propose to use this automatically generated mask on a second pass to mask the
image tokens, thus restricting them to the subject, not the background,
allowing the text prompt to attend to the rest of the image. For text prompts
describing locations and places, this produces images that accurately depict
the subject while definitively matching the prompt. We compare our method to a
few other test time personalization methods, and find our method displays high
prompt and source image alignment.
Using an automatically generated mask to restrict image tokens during inference improves prompt and source image alignment in personalized diffusion models.
diffusion models IP-Adapter masks image tokens prompt and source image alignment