IMAGINE-E: Image Generation Intelligence Evaluation of State-of-the-art
  Text-to-Image Models
With the rapid development of diffusion models, text-to-image(T2I) models
have made significant progress, showcasing impressive abilities in prompt
following and image generation. Recently launched models such as FLUX.1 and
Ideogram2.0, along with others like Dall-E3 and Stable Diffusion 3, have
demonstrated exceptional performance across various complex tasks, raising
questions about whether T2I models are moving towards general-purpose
applicability. Beyond traditional image generation, these models exhibit
capabilities across a range of fields, including controllable generation, image
editing, video, audio, 3D, and motion generation, as well as computer vision
tasks like semantic segmentation and depth estimation. However, current
evaluation frameworks are insufficient to comprehensively assess these models'
performance across expanding domains. To thoroughly evaluate these models, we
developed the IMAGINE-E and tested six prominent models: FLUX.1, Ideogram2.0,
Midjourney, Dall-E3, Stable Diffusion 3, and Jimeng. Our evaluation is divided
into five key domains: structured output generation, realism, and physical
consistency, specific domain generation, challenging scenario generation, and
multi-style creation tasks. This comprehensive assessment highlights each
model's strengths and limitations, particularly the outstanding performance of
FLUX.1 and Ideogram2.0 in structured and specific domain tasks, underscoring
the expanding applications and potential of T2I models as foundational AI
tools. This study provides valuable insights into the current state and future
trajectory of T2I models as they evolve towards general-purpose usability.
Evaluation scripts will be released at https://github.com/jylei16/Imagine-e.
679316ff3698fd97252a8e6f
Jiayi Lei
admin_assigned
2025-01-24T10:12:22.641Z
64c3c72e8f31d1e6c664b052
/avatars/af1ad5048eaa9dc417837ad02f927911.svg
jiayi lei
user
jyjyjyjy
679316ff3698fd97252a8e70
Renrui Zhang
679316ff3698fd97252a8e71
Xiangfei Hu
679316ff3698fd97252a8e72
Weifeng Lin
admin_assigned
2025-01-24T10:13:07.303Z
66026c9068d519ed32519e9c
/avatars/8fa051312c713772e5b8ba65989ff7f5.svg
Weifeng Lin
user
Afeng-x
679316ff3698fd97252a8e73
Zhen Li
claimed_verified
2025-01-24T20:51:38.124Z
6285a9133ab6642179158944
/avatars/6e10fa07c94141fcdbe0cab02bb731ca.svg
Zhen Li
user
Paper99
679316ff3698fd97252a8e74
Wenjian Sun
679316ff3698fd97252a8e75
Ruoyi Du
admin_assigned
2025-01-24T10:13:21.861Z
64a54586c0f13de8e7093314
/avatars/389e43e9a32cf2fc95f8f3a23b8f0508.svg
Ruoyi Du
user
RuoyiDu
679316ff3698fd97252a8e76
Le Zhuo
admin_assigned
2025-01-24T10:13:27.523Z
6358a167f56b03ec9147074d
/avatars/e54ea7bf0c240cf76d538296efb3976c.svg
Le Zhuo
user
JackyZhuo
679316ff3698fd97252a8e77
Zhongyu Li
admin_assigned
2025-01-24T10:13:33.108Z
6740a5730bb4a675446a80ad
/avatars/27c08e33df88e4f73c136da65f2b5adb.svg
Zhong-Yu Li
user
lzyhha
679316ff3698fd97252a8e78
Xinyue Li
claimed_verified
2025-03-28T08:38:51.843Z
66aba287b0f0b7411f511a47
/avatars/1450f182c38e80066ae5ea5df4fa218f.svg
Xinyue Li
user
Xxxy13
679316ff3698fd97252a8e79
Shitian Zhao
claimed_verified
2025-01-24T13:30:14.688Z
62c66504031996c36c86976a
https://cdn-avatars.huggingface.co/v1/production/uploads/62c66504031996c36c86976a/wIq0YJhkWnEhlzsh-TGYO.png
steve z
user
stzhao
679316ff3698fd97252a8e7a
Ziyu Guo
admin_assigned
2025-01-24T10:14:21.821Z
647d9ab61a1fcad2fdbf2d3d
/avatars/48c8aeae8979d2c87df8bde922437d62.svg
Ziyu Guo
user
ZiyuG
679316ff3698fd97252a8e7b
Yiting Lu
admin_assigned
2025-01-24T10:14:50.714Z
6614fb3d5aed02b298a4b469
/avatars/d0ddb4f989ad1a3f24128cc843347bde.svg
yiting lu
user
yeeeeeyy
679316ff3698fd97252a8e7c
Peng Gao
admin_assigned
2025-01-24T10:15:04.489Z
6759af3eccbc8817f9169179
/avatars/49e64c7ccf71b8f25c52783b6ae93620.svg
Peng Gao
user
gaopenghigh
679316ff3698fd97252a8e7d
Hongsheng Li
admin_assigned
2025-01-24T10:15:11.437Z
65c04e9c27a5fdca81abcbd9
/avatars/12a155683c824fa23da4a9e2bed4f64e.svg
Hongsheng LI
user
hsli-cuhk
IMAGINE-E: Image Generation Intelligence Evaluation of State-of-the-art
  Text-to-Image Models
With the rapid development of diffusion models, text-to-image(T2I) models
have made significant progress, showcasing impressive abilities in prompt
following and image generation. Recently launched models such as FLUX.1 and
Ideogram2.0, along with others like Dall-E3 and Stable Diffusion 3, have
demonstrated exceptional performance across various complex tasks, raising
questions about whether T2I models are moving towards general-purpose
applicability. Beyond traditional image generation, these models exhibit
capabilities across a range of fields, including controllable generation, image
editing, video, audio, 3D, and motion generation, as well as computer vision
tasks like semantic segmentation and depth estimation. However, current
evaluation frameworks are insufficient to comprehensively assess these models'
performance across expanding domains. To thoroughly evaluate these models, we
developed the IMAGINE-E and tested six prominent models: FLUX.1, Ideogram2.0,
Midjourney, Dall-E3, Stable Diffusion 3, and Jimeng. Our evaluation is divided
into five key domains: structured output generation, realism, and physical
consistency, specific domain generation, challenging scenario generation, and
multi-style creation tasks. This comprehensive assessment highlights each
model's strengths and limitations, particularly the outstanding performance of
FLUX.1 and Ideogram2.0 in structured and specific domain tasks, underscoring
the expanding applications and potential of T2I models as foundational AI
tools. This study provides valuable insights into the current state and future
trajectory of T2I models as they evolve towards general-purpose usability.
Evaluation scripts will be released at https://github.com/jylei16/Imagine-e.
IMAGINE-E evaluates multiple text-to-image models across various domains, highlighting their strengths and limitations, and underscores the expanding applicability of T2I models.
diffusion models text-to-image T2I FLUX.1 Ideogram2.0 Dall-E3 Stable Diffusion 3 controllable generation image editing video audio 3D motion generation semantic segmentation depth estimation structured output generation realism physical consistency specific domain generation challenging scenario generation multi-style creation tasks