Constantly Improving Image Models Need Constantly Improving Benchmarks
Recent advances in image generation, often driven by proprietary systems like
GPT-4o Image Gen, regularly introduce new capabilities that reshape how users
interact with these models. Existing benchmarks often lag behind and fail to
capture these emerging use cases, leaving a gap between community perceptions
of progress and formal evaluation. To address this, we present ECHO, a
framework for constructing benchmarks directly from real-world evidence of
model use: social media posts that showcase novel prompts and qualitative user
judgments. Applying this framework to GPT-4o Image Gen, we construct a dataset
of over 31,000 prompts curated from such posts. Our analysis shows that ECHO
(1) discovers creative and complex tasks absent from existing benchmarks, such
as re-rendering product labels across languages or generating receipts with
specified totals, (2) more clearly distinguishes state-of-the-art models from
alternatives, and (3) surfaces community feedback that we use to inform the
design of metrics for model quality (e.g., measuring observed shifts in color,
identity, and structure). Our website is at https://echo-bench.github.io.
68f58fc18589920bf4d32065
Jiaxin Ge
68f58fc18589920bf4d32066
Grace Luo
68f58fc18589920bf4d32067
Heekyung Lee
68f58fc18589920bf4d32068
Nishant Malpani
68f58fc18589920bf4d32069
Long Lian
68f58fc18589920bf4d3206a
XuDong Wang
68f58fc18589920bf4d3206b
Aleksander Holynski
68f58fc18589920bf4d3206c
Trevor Darrell
68f58fc18589920bf4d3206d
Sewon Min
68f58fc18589920bf4d3206e
David M. Chan
Constantly Improving Image Models Need Constantly Improving Benchmarks
Recent advances in image generation, often driven by proprietary systems like
GPT-4o Image Gen, regularly introduce new capabilities that reshape how users
interact with these models. Existing benchmarks often lag behind and fail to
capture these emerging use cases, leaving a gap between community perceptions
of progress and formal evaluation. To address this, we present ECHO, a
framework for constructing benchmarks directly from real-world evidence of
model use: social media posts that showcase novel prompts and qualitative user
judgments. Applying this framework to GPT-4o Image Gen, we construct a dataset
of over 31,000 prompts curated from such posts. Our analysis shows that ECHO
(1) discovers creative and complex tasks absent from existing benchmarks, such
as re-rendering product labels across languages or generating receipts with
specified totals, (2) more clearly distinguishes state-of-the-art models from
alternatives, and (3) surfaces community feedback that we use to inform the
design of metrics for model quality (e.g., measuring observed shifts in color,
identity, and structure). Our website is at https://echo-bench.github.io.
ECHO is a framework that constructs benchmarks for image generation models using real-world social media data, uncovering complex tasks and improving model evaluation.
image generation GPT-4o Image Gen benchmarks social media posts qualitative user judgments dataset re-rendering product labels generating receipts model quality metrics color shifts identity shifts structure shifts