Rethinking FID: Towards a Better Evaluation Metric for Image Generation
As with many machine learning problems, the progress of image generation
methods hinges on good evaluation metrics. One of the most popular is the
Frechet Inception Distance (FID). FID estimates the distance between a
distribution of Inception-v3 features of real images, and those of images
generated by the algorithm. We highlight important drawbacks of FID:
Inception's poor representation of the rich and varied content generated by
modern text-to-image models, incorrect normality assumptions, and poor sample
complexity. We call for a reevaluation of FID's use as the primary quality
metric for generated images. We empirically demonstrate that FID contradicts
human raters, it does not reflect gradual improvement of iterative
text-to-image models, it does not capture distortion levels, and that it
produces inconsistent results when varying the sample size. We also propose an
alternative new metric, CMMD, based on richer CLIP embeddings and the maximum
mean discrepancy distance with the Gaussian RBF kernel. It is an unbiased
estimator that does not make any assumptions on the probability distribution of
the embeddings and is sample efficient. Through extensive experiments and
analysis, we demonstrate that FID-based evaluations of text-to-image models may
be unreliable, and that CMMD offers a more robust and reliable assessment of
image quality.
65a9e617e0ee7990a6ed4e3c
Sadeep Jayasumana
admin_assigned
2024-01-19T13:43:07.027Z
65393b586882a4dcf7bba384
/avatars/fb8e808b3b526cc9af4ac625b1935088.svg
Sadeep Jayasumana
user
sadeepj
65a9e617e0ee7990a6ed4e3d
Srikumar Ramalingam
admin_assigned
2024-01-19T13:43:13.050Z
64916f233da8270af8580eb1
/avatars/40100ef7f72c8233f40713264bf39fc5.svg
Srikumar Ramalingam
user
srikumar
65a9e617e0ee7990a6ed4e3e
Andreas Veit
65a9e617e0ee7990a6ed4e3f
Daniel Glasner
65a9e617e0ee7990a6ed4e40
Ayan Chakrabarti
65a9e617e0ee7990a6ed4e41
Sanjiv Kumar
Rethinking FID: Towards a Better Evaluation Metric for Image Generation
As with many machine learning problems, the progress of image generation
methods hinges on good evaluation metrics. One of the most popular is the
Frechet Inception Distance (FID). FID estimates the distance between a
distribution of Inception-v3 features of real images, and those of images
generated by the algorithm. We highlight important drawbacks of FID:
Inception's poor representation of the rich and varied content generated by
modern text-to-image models, incorrect normality assumptions, and poor sample
complexity. We call for a reevaluation of FID's use as the primary quality
metric for generated images. We empirically demonstrate that FID contradicts
human raters, it does not reflect gradual improvement of iterative
text-to-image models, it does not capture distortion levels, and that it
produces inconsistent results when varying the sample size. We also propose an
alternative new metric, CMMD, based on richer CLIP embeddings and the maximum
mean discrepancy distance with the Gaussian RBF kernel. It is an unbiased
estimator that does not make any assumptions on the probability distribution of
the embeddings and is sample efficient. Through extensive experiments and
analysis, we demonstrate that FID-based evaluations of text-to-image models may
be unreliable, and that CMMD offers a more robust and reliable assessment of
image quality.
The paper identifies and critiques limitations of FID as an evaluation metric for image generation, proposing a new metric, CMMD, based on CLIP embeddings and maximum mean discrepancy for more reliable evaluations.
Frechet Inception Distance Inception-v3 text-to-image models CLIP embeddings maximum mean discrepancy Gaussian RBF kernel