ImagenHub: Standardizing the evaluation of conditional image generation
  models
Recently, a myriad of conditional image generation and editing models have
been developed to serve different downstream tasks, including text-to-image
generation, text-guided image editing, subject-driven image generation,
control-guided image generation, etc. However, we observe huge inconsistencies
in experimental conditions: datasets, inference, and evaluation metrics -
render fair comparisons difficult. This paper proposes ImagenHub, which is a
one-stop library to standardize the inference and evaluation of all the
conditional image generation models. Firstly, we define seven prominent tasks
and curate high-quality evaluation datasets for them. Secondly, we built a
unified inference pipeline to ensure fair comparison. Thirdly, we design two
human evaluation scores, i.e. Semantic Consistency and Perceptual Quality,
along with comprehensive guidelines to evaluate generated images. We train
expert raters to evaluate the model outputs based on the proposed metrics. Our
human evaluation achieves a high inter-worker agreement of Krippendorff's alpha
on 76% models with a value higher than 0.4. We comprehensively evaluated a
total of around 30 models and observed three key takeaways: (1) the existing
models' performance is generally unsatisfying except for Text-guided Image
Generation and Subject-driven Image Generation, with 74% models achieving an
overall score lower than 0.5. (2) we examined the claims from published papers
and found 83% of them hold with a few exceptions. (3) None of the existing
automatic metrics has a Spearman's correlation higher than 0.2 except
subject-driven image generation. Moving forward, we will continue our efforts
to evaluate newly published models and update our leaderboard to keep track of
the progress in conditional image generation.
651cdb74312b6fd1c6eb6eec
Max Ku
claimed_verified
2023-10-05T06:34:04.408Z
631d760344503b7227837242
https://cdn-avatars.huggingface.co/v1/production/uploads/631d760344503b7227837242/3b6JRusFX6GKJpsN9ZdeJ.png
Max Ku
user
vinesmsuic
651cdb74312b6fd1c6eb6eed
Tianle Li
651cdb74312b6fd1c6eb6eee
Kai Zhang
claimed_verified
2023-11-30T09:30:37.834Z
63e0a50242591dda0b9dca5c
https://cdn-avatars.huggingface.co/v1/production/uploads/63e0a50242591dda0b9dca5c/c7cBPEBWQDFYimfGnO_SI.png
Kai Zhang
user
drogozhang
651cdb74312b6fd1c6eb6eef
Yujie Lu
claimed_verified
2024-06-05T07:31:06.369Z
633cfb6ad5935998f7582b7b
https://cdn-avatars.huggingface.co/v1/production/uploads/633cfb6ad5935998f7582b7b/mzU4lI_GWPsYOFZHTBsYA.jpeg
Yujie Lu
user
FunCube
651cdb74312b6fd1c6eb6ef0
Xingyu Fu
claimed_verified
2024-04-19T07:39:45.830Z
6336091b2db86a181ccd6054
/avatars/829f69436225d05d2c2136bc90f640d7.svg
Xingyu Fu
user
Fiaa
651cdb74312b6fd1c6eb6ef1
Wenwen Zhuang
claimed_verified
2023-10-11T07:35:08.258Z
642f83089b2484d7d8568787
/avatars/5c502332d23d7332b23fc45c704782ac.svg
Wenwen Zhuang
user
Anwen0809
651cdb74312b6fd1c6eb6ef2
Wenhu Chen
ImagenHub: Standardizing the evaluation of conditional image generation
  models
Recently, a myriad of conditional image generation and editing models have
been developed to serve different downstream tasks, including text-to-image
generation, text-guided image editing, subject-driven image generation,
control-guided image generation, etc. However, we observe huge inconsistencies
in experimental conditions: datasets, inference, and evaluation metrics -
render fair comparisons difficult. This paper proposes ImagenHub, which is a
one-stop library to standardize the inference and evaluation of all the
conditional image generation models. Firstly, we define seven prominent tasks
and curate high-quality evaluation datasets for them. Secondly, we built a
unified inference pipeline to ensure fair comparison. Thirdly, we design two
human evaluation scores, i.e. Semantic Consistency and Perceptual Quality,
along with comprehensive guidelines to evaluate generated images. We train
expert raters to evaluate the model outputs based on the proposed metrics. Our
human evaluation achieves a high inter-worker agreement of Krippendorff's alpha
on 76% models with a value higher than 0.4. We comprehensively evaluated a
total of around 30 models and observed three key takeaways: (1) the existing
models' performance is generally unsatisfying except for Text-guided Image
Generation and Subject-driven Image Generation, with 74% models achieving an
overall score lower than 0.5. (2) we examined the claims from published papers
and found 83% of them hold with a few exceptions. (3) None of the existing
automatic metrics has a Spearman's correlation higher than 0.2 except
subject-driven image generation. Moving forward, we will continue our efforts
to evaluate newly published models and update our leaderboard to keep track of
the progress in conditional image generation.
ImagenHub standardizes inference and evaluation for conditional image generation models, revealing inconsistencies and highlighting areas needing improvement.
conditional image generation text-to-image generation text-guided image editing subject-driven image generation control-guided image generation Krippendorff's alpha semantic consistency perceptual quality