Training-free Regional Prompting for Diffusion Transformers
Diffusion models have demonstrated excellent capabilities in text-to-image
generation. Their semantic understanding (i.e., prompt following) ability has
also been greatly improved with large language models (e.g., T5, Llama).
However, existing models cannot perfectly handle long and complex text prompts,
especially when the text prompts contain various objects with numerous
attributes and interrelated spatial relationships. While many regional
prompting methods have been proposed for UNet-based models (SD1.5, SDXL), but
there are still no implementations based on the recent Diffusion Transformer
(DiT) architecture, such as SD3 and FLUX.1.In this report, we propose and
implement regional prompting for FLUX.1 based on attention manipulation, which
enables DiT with fined-grained compositional text-to-image generation
capability in a training-free manner. Code is available at
https://github.com/antonioo-c/Regional-Prompting-FLUX.
6729ba9886503ffe1cc31897
Anthony Chen
claimed_verified
2024-11-05T07:57:18.751Z
6311d9ee04f842f79916158c
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/PvOYgDEpe0x5ExLyj1BK-.png
chen
user
antonio-c
6729ba9886503ffe1cc31898
Jianjin Xu
admin_assigned
2024-11-05T13:59:09.353Z
6323c5a899c4839e1c6223e9
/avatars/7890aa86a309e2c86e7341212f8afc40.svg
Jianjin Xu
user
atlantix
6729ba9886503ffe1cc31899
Wenzhao Zheng
6729ba9886503ffe1cc3189a
Gaole Dai
admin_assigned
2024-11-05T13:59:23.394Z
668d9de515511820b9c31a51
/avatars/409f1334969f659a331a61bb0f4ce15c.svg
Gaole Dai
user
barkleydai
6729ba9886503ffe1cc3189b
Yida Wang
claimed_verified
2024-11-05T07:57:13.829Z
647068944be5cf1f33491cb9
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Aqudl2PTSKPAylBhlccWr.png
Yida Wang
user
wangyida
6729ba9886503ffe1cc3189c
Renrui Zhang
6729ba9886503ffe1cc3189d
Haofan Wang
claimed_verified
2024-11-05T07:57:15.554Z
637745113a63a2983ffbde13
https://cdn-avatars.huggingface.co/v1/production/uploads/1669187672174-637745113a63a2983ffbde13.jpeg
Haofan Wang
user
wanghaofan
6729ba9886503ffe1cc3189e
Shanghang Zhang
Training-free Regional Prompting for Diffusion Transformers
Diffusion models have demonstrated excellent capabilities in text-to-image
generation. Their semantic understanding (i.e., prompt following) ability has
also been greatly improved with large language models (e.g., T5, Llama).
However, existing models cannot perfectly handle long and complex text prompts,
especially when the text prompts contain various objects with numerous
attributes and interrelated spatial relationships. While many regional
prompting methods have been proposed for UNet-based models (SD1.5, SDXL), but
there are still no implementations based on the recent Diffusion Transformer
(DiT) architecture, such as SD3 and FLUX.1.In this report, we propose and
implement regional prompting for FLUX.1 based on attention manipulation, which
enables DiT with fined-grained compositional text-to-image generation
capability in a training-free manner. Code is available at
https://github.com/antonioo-c/Regional-Prompting-FLUX.
Regional prompting is implemented for the FLUX diffusion model using attention manipulation to achieve fine-grained text-to-image generation without training.
diffusion models text-to-image generation large language models T5 Llama regional prompting UNet Diffusion Transformer DiT attention manipulation fined-grained compositional text-to-image generation