SeeGULL: A Stereotype Benchmark with Broad Geo-Cultural Coverage
  Leveraging Generative Models
Stereotype benchmark datasets are crucial to detect and mitigate social
stereotypes about groups of people in NLP models. However, existing datasets
are limited in size and coverage, and are largely restricted to stereotypes
prevalent in the Western society. This is especially problematic as language
technologies gain hold across the globe. To address this gap, we present
SeeGULL, a broad-coverage stereotype dataset, built by utilizing generative
capabilities of large language models such as PaLM, and GPT-3, and leveraging a
globally diverse rater pool to validate the prevalence of those stereotypes in
society. SeeGULL is in English, and contains stereotypes about identity groups
spanning 178 countries across 8 different geo-political regions across 6
continents, as well as state-level identities within the US and India. We also
include fine-grained offensiveness scores for different stereotypes and
demonstrate their global disparities. Furthermore, we include comparative
annotations about the same groups by annotators living in the region vs. those
that are based in North America, and demonstrate that within-region stereotypes
about groups differ from those prevalent in North America. CONTENT WARNING:
This paper contains stereotype examples that may be offensive.
646aeaf3a5dd10c9a4a2ff86
Akshita Jha
extracted_pending
2023-05-24T17:27:04.949Z
646e48d80867c99c2d3dfabb
/avatars/9f34ada512447823a8718cfd4d5bfd16.svg
Akshita Jha
user
akshitajha
646aeaf3a5dd10c9a4a2ff87
Aida Davani
extracted_pending
2025-05-19T18:48:09.297Z
682b7cce8f72754097083a48
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/yEKYS5pT2HxXHHERCsBu7.png
Aida Davani
user
aidamd
646aeaf3a5dd10c9a4a2ff88
Chandan K. Reddy
646aeaf3a5dd10c9a4a2ff89
Shachi Dave
646aeaf3a5dd10c9a4a2ff8a
Vinodkumar Prabhakaran
extracted_pending
2023-05-22T04:09:24.147Z
60c2b4cf22a9d3ec6810297b
/avatars/52d8fbda4233da69e406814c278dca02.svg
Vinodkumar Prabhakaran
user
vinodkpg
646aeaf3a5dd10c9a4a2ff8b
Sunipa Dev
extracted_pending
2023-05-22T04:09:24.147Z
62e82ec840bd989bb71853ec
/avatars/c194747a796a2d61e6e5e793f9fefb2d.svg
Sunipa Dev
user
sunipadev
SeeGULL: A Stereotype Benchmark with Broad Geo-Cultural Coverage
  Leveraging Generative Models
Stereotype benchmark datasets are crucial to detect and mitigate social
stereotypes about groups of people in NLP models. However, existing datasets
are limited in size and coverage, and are largely restricted to stereotypes
prevalent in the Western society. This is especially problematic as language
technologies gain hold across the globe. To address this gap, we present
SeeGULL, a broad-coverage stereotype dataset, built by utilizing generative
capabilities of large language models such as PaLM, and GPT-3, and leveraging a
globally diverse rater pool to validate the prevalence of those stereotypes in
society. SeeGULL is in English, and contains stereotypes about identity groups
spanning 178 countries across 8 different geo-political regions across 6
continents, as well as state-level identities within the US and India. We also
include fine-grained offensiveness scores for different stereotypes and
demonstrate their global disparities. Furthermore, we include comparative
annotations about the same groups by annotators living in the region vs. those
that are based in North America, and demonstrate that within-region stereotypes
about groups differ from those prevalent in North America. CONTENT WARNING:
This paper contains stereotype examples that may be offensive.
SeeGULL is a globally diverse stereotype dataset built using large language models, spanning 178 countries and including offensiveness scores and regional comparisons.
generative capabilities large language models stereotype dataset globally diverse offensiveness scores regional comparisons