FRAP: Faithful and Realistic Text-to-Image Generation with Adaptive
  Prompt Weighting
Text-to-image (T2I) diffusion models have demonstrated impressive
capabilities in generating high-quality images given a text prompt. However,
ensuring the prompt-image alignment remains a considerable challenge, i.e.,
generating images that faithfully align with the prompt's semantics. Recent
works attempt to improve the faithfulness by optimizing the latent code, which
potentially could cause the latent code to go out-of-distribution and thus
produce unrealistic images. In this paper, we propose FRAP, a simple, yet
effective approach based on adaptively adjusting the per-token prompt weights
to improve prompt-image alignment and authenticity of the generated images. We
design an online algorithm to adaptively update each token's weight
coefficient, which is achieved by minimizing a unified objective function that
encourages object presence and the binding of object-modifier pairs. Through
extensive evaluations, we show FRAP generates images with significantly higher
prompt-image alignment to prompts from complex datasets, while having a lower
average latency compared to recent latent code optimization methods, e.g., 4
seconds faster than D&B on the COCO-Subject dataset. Furthermore, through
visual comparisons and evaluation on the CLIP-IQA-Real metric, we show that
FRAP not only improves prompt-image alignment but also generates more authentic
images with realistic appearances. We also explore combining FRAP with prompt
rewriting LLM to recover their degraded prompt-image alignment, where we
observe improvements in both prompt-image alignment and image quality.
66c73909a575572fcb8d3cf5
Liyao Jiang
claimed_verified
2024-08-22T19:50:14.732Z
64ac49ccb7d86b40fd60a8dd
/avatars/e9f5482cffdd1d5917523a496a3805f0.svg
Liyao Jiang
user
LiyaoJiang
66c73909a575572fcb8d3cf6
Negar Hassanpour
66c73909a575572fcb8d3cf7
Mohammad Salameh
66c73909a575572fcb8d3cf8
Mohan Sai Singamsetti
66c73909a575572fcb8d3cf9
Fengyu Sun
66c73909a575572fcb8d3cfa
Wei Lu
66c73909a575572fcb8d3cfb
Di Niu
FRAP: Faithful and Realistic Text-to-Image Generation with Adaptive
  Prompt Weighting
Text-to-image (T2I) diffusion models have demonstrated impressive
capabilities in generating high-quality images given a text prompt. However,
ensuring the prompt-image alignment remains a considerable challenge, i.e.,
generating images that faithfully align with the prompt's semantics. Recent
works attempt to improve the faithfulness by optimizing the latent code, which
potentially could cause the latent code to go out-of-distribution and thus
produce unrealistic images. In this paper, we propose FRAP, a simple, yet
effective approach based on adaptively adjusting the per-token prompt weights
to improve prompt-image alignment and authenticity of the generated images. We
design an online algorithm to adaptively update each token's weight
coefficient, which is achieved by minimizing a unified objective function that
encourages object presence and the binding of object-modifier pairs. Through
extensive evaluations, we show FRAP generates images with significantly higher
prompt-image alignment to prompts from complex datasets, while having a lower
average latency compared to recent latent code optimization methods, e.g., 4
seconds faster than D&B on the COCO-Subject dataset. Furthermore, through
visual comparisons and evaluation on the CLIP-IQA-Real metric, we show that
FRAP not only improves prompt-image alignment but also generates more authentic
images with realistic appearances. We also explore combining FRAP with prompt
rewriting LLM to recover their degraded prompt-image alignment, where we
observe improvements in both prompt-image alignment and image quality.
FRAP enhances text-to-image alignment and authenticity by adaptively adjusting per-token prompt weights, improving image quality and reducing latency compared to latent code optimization methods.
text-to-image diffusion models prompt-image alignment latent code FRAP per-token prompt weights online algorithm unified objective function object presence object-modifier pairs COCO-Subject CLIP-IQA-Real prompt rewriting LLM