ComfyGen: Prompt-Adaptive Workflows for Text-to-Image Generation
The practical use of text-to-image generation has evolved from simple,
monolithic models to complex workflows that combine multiple specialized
components. While workflow-based approaches can lead to improved image quality,
crafting effective workflows requires significant expertise, owing to the large
number of available components, their complex inter-dependence, and their
dependence on the generation prompt. Here, we introduce the novel task of
prompt-adaptive workflow generation, where the goal is to automatically tailor
a workflow to each user prompt. We propose two LLM-based approaches to tackle
this task: a tuning-based method that learns from user-preference data, and a
training-free method that uses the LLM to select existing flows. Both
approaches lead to improved image quality when compared to monolithic models or
generic, prompt-independent workflows. Our work shows that prompt-dependent
flow prediction offers a new pathway to improving text-to-image generation
quality, complementing existing research directions in the field.
66fe0d6064acd6a2de1864b8
Rinon Gal
claimed_verified
2024-10-03T09:14:20.320Z
627c1360f19c5eb46d55ba05
https://cdn-avatars.huggingface.co/v1/production/uploads/1652712871747-627c1360f19c5eb46d55ba05.jpeg
Rinon Gal
user
rinong
66fe0d6064acd6a2de1864b9
Adi Haviv
admin_assigned
2024-10-03T08:52:49.297Z
610b89ccf5098560ee5fa698
https://cdn-avatars.huggingface.co/v1/production/uploads/1628156505064-610b89ccf5098560ee5fa698.jpeg
Adi Haviv
user
adihaviv
66fe0d6064acd6a2de1864ba
Yuval Alaluf
admin_assigned
2024-10-03T08:52:41.386Z
6310c1e0e87051f3e3e3bcc1
/avatars/61b24a2eace7da0e27fbc4369a046051.svg
Yuval Alaluf
user
yuvalalaluf
66fe0d6064acd6a2de1864bb
Amit H. Bermano
66fe0d6064acd6a2de1864bc
Daniel Cohen-Or
admin_assigned
2024-10-03T08:52:28.974Z
628507161949ebcae8e24ec3
/avatars/008ecb3daa4c8187b5f339f1176b3c39.svg
Daniel Cohen-Or
user
cohenor
66fe0d6064acd6a2de1864bd
Gal Chechik
admin_assigned
2024-10-03T08:52:23.222Z
6493393f357b252af72196c5
https://cdn-avatars.huggingface.co/v1/production/uploads/6493393f357b252af72196c5/EWSy18XRcMRa_4XMM3Fu-.jpeg
Gal Chechik
user
galchechik
ComfyGen: Prompt-Adaptive Workflows for Text-to-Image Generation
The practical use of text-to-image generation has evolved from simple,
monolithic models to complex workflows that combine multiple specialized
components. While workflow-based approaches can lead to improved image quality,
crafting effective workflows requires significant expertise, owing to the large
number of available components, their complex inter-dependence, and their
dependence on the generation prompt. Here, we introduce the novel task of
prompt-adaptive workflow generation, where the goal is to automatically tailor
a workflow to each user prompt. We propose two LLM-based approaches to tackle
this task: a tuning-based method that learns from user-preference data, and a
training-free method that uses the LLM to select existing flows. Both
approaches lead to improved image quality when compared to monolithic models or
generic, prompt-independent workflows. Our work shows that prompt-dependent
flow prediction offers a new pathway to improving text-to-image generation
quality, complementing existing research directions in the field.
Automatic generation of text-to-image workflows based on user prompts enhances image quality compared to monolithic models or generic workflows.
prompt-adaptive workflow generation LLM-based approaches tuning-based method training-free method prompt-dependent flow prediction