ModelCitizens: Representing Community Voices in Online Safety
Automatic toxic language detection is critical for creating safe, inclusive
online spaces. However, it is a highly subjective task, with perceptions of
toxic language shaped by community norms and lived experience. Existing
toxicity detection models are typically trained on annotations that collapse
diverse annotator perspectives into a single ground truth, erasing important
context-specific notions of toxicity such as reclaimed language. To address
this, we introduce MODELCITIZENS, a dataset of 6.8K social media posts and 40K
toxicity annotations across diverse identity groups. To capture the role of
conversational context on toxicity, typical of social media posts, we augment
MODELCITIZENS posts with LLM-generated conversational scenarios.
State-of-the-art toxicity detection tools (e.g. OpenAI Moderation API,
GPT-o4-mini) underperform on MODELCITIZENS, with further degradation on
context-augmented posts. Finally, we release LLAMACITIZEN-8B and
GEMMACITIZEN-12B, LLaMA- and Gemma-based models finetuned on MODELCITIZENS,
which outperform GPT-o4-mini by 5.5% on in-distribution evaluations. Our
findings highlight the importance of community-informed annotation and modeling
for inclusive content moderation. The data, models and code are available at
https://github.com/asuvarna31/modelcitizens.
686f5ec6d938c25d68441bc1
Ashima Suvarna
claimed_verified
2025-07-10T09:12:39.834Z
628c29a54c5a62a1d216c560
/avatars/d21b4da766f87f47228112958666643b.svg
Ashima Suvarna
user
Ashima
686f5ec6d938c25d68441bc2
Christina Chance
admin_assigned
2025-07-10T09:16:25.997Z
6543d269326cb9a32bd58e40
/avatars/ddabbd80844e3dff676a8c2a182d920c.svg
Christina
user
christinachance
686f5ec6d938c25d68441bc3
Karolina Naranjo
686f5ec6d938c25d68441bc4
Hamid Palangi
admin_assigned
2025-07-10T09:16:36.140Z
6189ddd0992df2640e3e7d40
/avatars/925f2308fc412ae352b57a1b71815028.svg
Hamid Palangi
user
hamidpalangi
686f5ec6d938c25d68441bc5
Sophie Hao
admin_assigned
2025-07-10T09:16:42.125Z
65d9787bec3dc9ccf7344f6f
/avatars/ab361193ed2287158f803fc792bb2df5.svg
Sophie Hao
user
notaphonologist
686f5ec6d938c25d68441bc6
Thomas Hartvigsen
686f5ec6d938c25d68441bc7
Saadia Gabriel
ModelCitizens: Representing Community Voices in Online Safety
Automatic toxic language detection is critical for creating safe, inclusive
online spaces. However, it is a highly subjective task, with perceptions of
toxic language shaped by community norms and lived experience. Existing
toxicity detection models are typically trained on annotations that collapse
diverse annotator perspectives into a single ground truth, erasing important
context-specific notions of toxicity such as reclaimed language. To address
this, we introduce MODELCITIZENS, a dataset of 6.8K social media posts and 40K
toxicity annotations across diverse identity groups. To capture the role of
conversational context on toxicity, typical of social media posts, we augment
MODELCITIZENS posts with LLM-generated conversational scenarios.
State-of-the-art toxicity detection tools (e.g. OpenAI Moderation API,
GPT-o4-mini) underperform on MODELCITIZENS, with further degradation on
context-augmented posts. Finally, we release LLAMACITIZEN-8B and
GEMMACITIZEN-12B, LLaMA- and Gemma-based models finetuned on MODELCITIZENS,
which outperform GPT-o4-mini by 5.5% on in-distribution evaluations. Our
findings highlight the importance of community-informed annotation and modeling
for inclusive content moderation. The data, models and code are available at
https://github.com/asuvarna31/modelcitizens.
MODELCITIZENS, a dataset with diverse toxicity annotations, and community-informed models outperform existing tools in detecting toxic language in social media posts, emphasizing the importance of context and community perspectives.
MODELCITIZENS LLM-generated conversational scenarios OpenAI Moderation API GPT-o4-mini LLAMACITIZEN-8B GEMMACITIZEN-12B LLaMA Gemma in-distribution evaluations