Interleaved Scene Graph for Interleaved Text-and-Image Generation
  Assessment
Many real-world user queries (e.g. "How do to make egg fried rice?") could
benefit from systems capable of generating responses with both textual steps
with accompanying images, similar to a cookbook. Models designed to generate
interleaved text and images face challenges in ensuring consistency within and
across these modalities. To address these challenges, we present ISG, a
comprehensive evaluation framework for interleaved text-and-image generation.
ISG leverages a scene graph structure to capture relationships between text and
image blocks, evaluating responses on four levels of granularity: holistic,
structural, block-level, and image-specific. This multi-tiered evaluation
allows for a nuanced assessment of consistency, coherence, and accuracy, and
provides interpretable question-answer feedback. In conjunction with ISG, we
introduce a benchmark, ISG-Bench, encompassing 1,150 samples across 8
categories and 21 subcategories. This benchmark dataset includes complex
language-vision dependencies and golden answers to evaluate models effectively
on vision-centric tasks such as style transfer, a challenging area for current
models. Using ISG-Bench, we demonstrate that recent unified vision-language
models perform poorly on generating interleaved content. While compositional
approaches that combine separate language and image models show a 111%
improvement over unified models at the holistic level, their performance
remains suboptimal at both block and image levels. To facilitate future work,
we develop ISG-Agent, a baseline agent employing a "plan-execute-refine"
pipeline to invoke tools, achieving a 122% performance improvement.
67467ee679406f42a1466a07
Dongping Chen
claimed_verified
2024-11-27T22:20:26.369Z
643be8879f5d314db2d9ed23
https://cdn-avatars.huggingface.co/v1/production/uploads/643be8879f5d314db2d9ed23/VrW2UtJ7ppOnGIYjTWd7b.png
Chen Dongping
user
shuaishuaicdp
67467ee679406f42a1466a08
Ruoxi Chen
admin_assigned
2024-11-28T10:39:29.970Z
65a135df18b7d7c225737173
/avatars/cc829bf663da9ca9df9d92feea80e42f.svg
Ruoxi Chen
user
Dipsy0830
67467ee679406f42a1466a09
Shu Pu
claimed_verified
2025-03-26T20:50:07.392Z
663ba53b96c98cfb8ded1c4f
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/6-ipDyZbMYj_lfVurLjvh.png
Shu Pu
user
pudashi
67467ee679406f42a1466a0a
Zhaoyi Liu
admin_assigned
2024-11-28T10:39:43.179Z
6695ca27bcd81f395ec16e6d
/avatars/6f70cd2fe6d8316dfcf555b3d8cf706c.svg
Zhaoyi LIU
user
lzy8465
67467ee679406f42a1466a0b
Yanru Wu
claimed_verified
2025-05-15T10:41:22.994Z
66ee4ec36babd2a70556b8e4
/avatars/d7b4c3ce1367e5b4ff8eab5647abbe0b.svg
YanruWu
user
YanruWu
67467ee679406f42a1466a0c
Caixi Chen
67467ee679406f42a1466a0d
Benlin Liu
admin_assigned
2024-11-28T10:39:59.992Z
6326723c21c98f83d6ad128e
/avatars/6c208ff815c037d0a6a2ada3198e7fd4.svg
Benlin Liu
user
Tim666
67467ee679406f42a1466a0e
Yue Huang
67467ee679406f42a1466a0f
Yao Wan
67467ee679406f42a1466a10
Pan Zhou
67467ee679406f42a1466a11
Ranjay Krishna
admin_assigned
2024-11-28T10:39:19.626Z
66429868ab89e3a3a85668b0
/avatars/170e0daa454838deee2bf946f7118651.svg
Ranjay Krishna
user
ranjaykrishna
Interleaved Scene Graph for Interleaved Text-and-Image Generation
  Assessment
Many real-world user queries (e.g. "How do to make egg fried rice?") could
benefit from systems capable of generating responses with both textual steps
with accompanying images, similar to a cookbook. Models designed to generate
interleaved text and images face challenges in ensuring consistency within and
across these modalities. To address these challenges, we present ISG, a
comprehensive evaluation framework for interleaved text-and-image generation.
ISG leverages a scene graph structure to capture relationships between text and
image blocks, evaluating responses on four levels of granularity: holistic,
structural, block-level, and image-specific. This multi-tiered evaluation
allows for a nuanced assessment of consistency, coherence, and accuracy, and
provides interpretable question-answer feedback. In conjunction with ISG, we
introduce a benchmark, ISG-Bench, encompassing 1,150 samples across 8
categories and 21 subcategories. This benchmark dataset includes complex
language-vision dependencies and golden answers to evaluate models effectively
on vision-centric tasks such as style transfer, a challenging area for current
models. Using ISG-Bench, we demonstrate that recent unified vision-language
models perform poorly on generating interleaved content. While compositional
approaches that combine separate language and image models show a 111%
improvement over unified models at the holistic level, their performance
remains suboptimal at both block and image levels. To facilitate future work,
we develop ISG-Agent, a baseline agent employing a "plan-execute-refine"
pipeline to invoke tools, achieving a 122% performance improvement.
ISG provides a comprehensive evaluation framework and benchmark for interleaved text-and-image generation, highlighting challenges and improvements over unified and compositional models.
interleaved text-and-image generation scene graph ISG ISG-Bench style transfer unified vision-language models compositional approaches ISG-Agent plan-execute-refine pipeline