World-To-Image: Grounding Text-to-Image Generation with Agent-Driven
  World Knowledge
While text-to-image (T2I) models can synthesize high-quality images, their
performance degrades significantly when prompted with novel or
out-of-distribution (OOD) entities due to inherent knowledge cutoffs. We
introduce World-To-Image, a novel framework that bridges this gap by empowering
T2I generation with agent-driven world knowledge. We design an agent that
dynamically searches the web to retrieve images for concepts unknown to the
base model. This information is then used to perform multimodal prompt
optimization, steering powerful generative backbones toward an accurate
synthesis. Critically, our evaluation goes beyond traditional metrics,
utilizing modern assessments like LLMGrader and ImageReward to measure true
semantic fidelity. Our experiments show that World-To-Image substantially
outperforms state-of-the-art methods in both semantic alignment and visual
aesthetics, achieving +8.1% improvement in accuracy-to-prompt on our curated
NICE benchmark. Our framework achieves these results with high efficiency in
less than three iterations, paving the way for T2I systems that can better
reflect the ever-changing real world. Our demo code is available
herehttps://github.com/mhson-kyle/World-To-Image.
68ecc496cd07fb414898ccad
Moo Hyun Son
68ecc496cd07fb414898ccae
Jintaek Oh
68ecc496cd07fb414898ccaf
Sun Bin Mun
68ecc496cd07fb414898ccb0
Jaechul Roh
68ecc496cd07fb414898ccb1
Sehyun Choi
World-To-Image: Grounding Text-to-Image Generation with Agent-Driven
  World Knowledge
While text-to-image (T2I) models can synthesize high-quality images, their
performance degrades significantly when prompted with novel or
out-of-distribution (OOD) entities due to inherent knowledge cutoffs. We
introduce World-To-Image, a novel framework that bridges this gap by empowering
T2I generation with agent-driven world knowledge. We design an agent that
dynamically searches the web to retrieve images for concepts unknown to the
base model. This information is then used to perform multimodal prompt
optimization, steering powerful generative backbones toward an accurate
synthesis. Critically, our evaluation goes beyond traditional metrics,
utilizing modern assessments like LLMGrader and ImageReward to measure true
semantic fidelity. Our experiments show that World-To-Image substantially
outperforms state-of-the-art methods in both semantic alignment and visual
aesthetics, achieving +8.1% improvement in accuracy-to-prompt on our curated
NICE benchmark. Our framework achieves these results with high efficiency in
less than three iterations, paving the way for T2I systems that can better
reflect the ever-changing real world. Our demo code is available
herehttps://github.com/mhson-kyle/World-To-Image.
World-To-Image enhances text-to-image generation by integrating web-based knowledge retrieval and multimodal prompt optimization, improving semantic accuracy and visual quality.
text-to-image out-of-distribution agent-driven world knowledge multimodal prompt optimization generative backbones LLMGrader ImageReward semantic fidelity NICE benchmark