FACET: Fairness in Computer Vision Evaluation Benchmark
Computer vision models have known performance disparities across attributes
such as gender and skin tone. This means during tasks such as classification
and detection, model performance differs for certain classes based on the
demographics of the people in the image. These disparities have been shown to
exist, but until now there has not been a unified approach to measure these
differences for common use-cases of computer vision models. We present a new
benchmark named FACET (FAirness in Computer Vision EvaluaTion), a large,
publicly available evaluation set of 32k images for some of the most common
vision tasks - image classification, object detection and segmentation. For
every image in FACET, we hired expert reviewers to manually annotate
person-related attributes such as perceived skin tone and hair type, manually
draw bounding boxes and label fine-grained person-related classes such as disk
jockey or guitarist. In addition, we use FACET to benchmark state-of-the-art
vision models and present a deeper understanding of potential performance
disparities and challenges across sensitive demographic attributes. With the
exhaustive annotations collected, we probe models using single demographics
attributes as well as multiple attributes using an intersectional approach
(e.g. hair color and perceived skin tone). Our results show that
classification, detection, segmentation, and visual grounding models exhibit
performance disparities across demographic attributes and intersections of
attributes. These harms suggest that not all people represented in datasets
receive fair and equitable treatment in these vision tasks. We hope current and
future results using our benchmark will contribute to fairer, more robust
vision models. FACET is available publicly at https://facet.metademolab.com/
64f531ba4937e64ca402e3ee
Laura Gustafson
64f531ba4937e64ca402e3ef
Chloe Rolland
admin_assigned
2023-09-04T08:19:34.205Z
6463d03f38083255f6754152
/avatars/a158e36e0a24ece399b8ec26a2084444.svg
Chloe Rolland
user
crolland
64f531ba4937e64ca402e3f0
Nikhila Ravi
64f531ba4937e64ca402e3f1
Quentin Duval
64f531ba4937e64ca402e3f2
Aaron Adcock
admin_assigned
2023-09-04T08:21:16.597Z
645bd1631c24cd669dd8f2b3
/avatars/f592e8b66a31ed70a745512b63926946.svg
Aaron Adcock
user
aadcock
64f531ba4937e64ca402e3f3
Cheng-Yang Fu
admin_assigned
2023-09-04T08:23:03.028Z
64cc01bd87bfb0fc2124b112
/avatars/32c20e200360e2bfd7bfe29ed37ea242.svg
Cheng-Yang Fu
user
chengyangfu1985
64f531ba4937e64ca402e3f4
Melissa Hall
claimed_verified
2024-07-18T09:11:08.621Z
6697bfb4acbced03ebd133f8
/avatars/a7d41367c6f7974303feb5af2a387ce4.svg
Melissa Hall
user
melissah14
64f531ba4937e64ca402e3f5
Candace Ross
admin_assigned
2023-09-04T08:23:30.473Z
628709d57cc3b7bc1b95fcf0
/avatars/98e0d9695527125a0409d1098cdceaa4.svg
Candace Ross
user
candaceross
FACET: Fairness in Computer Vision Evaluation Benchmark
Computer vision models have known performance disparities across attributes
such as gender and skin tone. This means during tasks such as classification
and detection, model performance differs for certain classes based on the
demographics of the people in the image. These disparities have been shown to
exist, but until now there has not been a unified approach to measure these
differences for common use-cases of computer vision models. We present a new
benchmark named FACET (FAirness in Computer Vision EvaluaTion), a large,
publicly available evaluation set of 32k images for some of the most common
vision tasks - image classification, object detection and segmentation. For
every image in FACET, we hired expert reviewers to manually annotate
person-related attributes such as perceived skin tone and hair type, manually
draw bounding boxes and label fine-grained person-related classes such as disk
jockey or guitarist. In addition, we use FACET to benchmark state-of-the-art
vision models and present a deeper understanding of potential performance
disparities and challenges across sensitive demographic attributes. With the
exhaustive annotations collected, we probe models using single demographics
attributes as well as multiple attributes using an intersectional approach
(e.g. hair color and perceived skin tone). Our results show that
classification, detection, segmentation, and visual grounding models exhibit
performance disparities across demographic attributes and intersections of
attributes. These harms suggest that not all people represented in datasets
receive fair and equitable treatment in these vision tasks. We hope current and
future results using our benchmark will contribute to fairer, more robust
vision models. FACET is available publicly at https://facet.metademolab.com/
A benchmark named FACET is introduced to evaluate performance disparities of computer vision models across demographic attributes through extensive annotations and intersectional analysis.
image classification object detection segmentation visual grounding manual annotations demographic attributes performance disparities benchmarks intersectional approach