Evaluating the Social Impact of Generative AI Systems in Systems and
  Society
Generative AI systems across modalities, ranging from text (including code),
image, audio, and video, have broad social impacts, but there is no official
standard for means of evaluating those impacts or for which impacts should be
evaluated. In this paper, we present a guide that moves toward a standard
approach in evaluating a base generative AI system for any modality in two
overarching categories: what can be evaluated in a base system independent of
context and what can be evaluated in a societal context. Importantly, this
refers to base systems that have no predetermined application or deployment
context, including a model itself, as well as system components, such as
training data. Our framework for a base system defines seven categories of
social impact: bias, stereotypes, and representational harms; cultural values
and sensitive content; disparate performance; privacy and data protection;
financial costs; environmental costs; and data and content moderation labor
costs. Suggested methods for evaluation apply to listed generative modalities
and analyses of the limitations of existing evaluations serve as a starting
point for necessary investment in future evaluations. We offer five overarching
categories for what can be evaluated in a broader societal context, each with
its own subcategories: trustworthiness and autonomy; inequality,
marginalization, and violence; concentration of authority; labor and
creativity; and ecosystem and environment. Each subcategory includes
recommendations for mitigating harm.
6486818b43ba3df95402d2cf
Irene Solaiman
extracted_confirmed
2023-06-12T04:26:55.608Z
62543749b777cd32720675c2
https://cdn-avatars.huggingface.co/v1/production/uploads/1658760912583-62543749b777cd32720675c2.jpeg
Irene Solaiman
user
irenesolaiman
6486818b43ba3df95402d2d0
Zeerak Talat
6486818b43ba3df95402d2d1
William Agnew
6486818b43ba3df95402d2d2
Lama Ahmad
6486818b43ba3df95402d2d3
Dylan Baker
6486818b43ba3df95402d2d4
Su Lin Blodgett
6486818b43ba3df95402d2d5
Hal Daumé III
6486818b43ba3df95402d2d6
Jesse Dodge
admin_assigned
2023-06-12T10:09:47.178Z
6283f38567d336d3e5d5280e
/avatars/d0a54aaec74a90b050e671c191b87a80.svg
Jesse Dodge
user
JesseDodge
6486818b43ba3df95402d2d7
Ellie Evans
admin_assigned
2023-06-12T10:10:45.057Z
626c0912974d6a67df700db6
/avatars/057a19f682e6a8d0411bc6efa25f2588.svg
Ellie Evans
user
ellieevans
6486818b43ba3df95402d2d8
Sara Hooker
admin_assigned
2023-06-12T10:11:04.380Z
63434eb76f59b79da07dbddf
https://cdn-avatars.huggingface.co/v1/production/uploads/63434eb76f59b79da07dbddf/BEwmVjqPNYlqmutXG0G6e.jpeg
Sara Hooker
user
sarahooker
6486818b43ba3df95402d2d9
Yacine Jernite
admin_assigned
2023-06-12T10:07:51.393Z
5ee3a7cd2a3eae3cbdad1305
https://cdn-avatars.huggingface.co/v1/production/uploads/1594144055859-5ee3a7cd2a3eae3cbdad1305.jpeg
Yacine Jernite
user
yjernite
6486818b43ba3df95402d2da
Alexandra Sasha Luccioni
admin_assigned
2023-06-12T10:08:05.485Z
60edd0133e2c73a9a21455f5
https://cdn-avatars.huggingface.co/v1/production/uploads/60edd0133e2c73a9a21455f5/yK1G-Fv-YjYb7v_chkz3p.jpeg
Sasha Luccioni
user
sasha
6486818b43ba3df95402d2db
Alberto Lusoli
6486818b43ba3df95402d2dc
Margaret Mitchell
admin_assigned
2023-06-12T10:08:18.780Z
60c757ea5f9a76ab3f844f12
https://cdn-avatars.huggingface.co/v1/production/uploads/1626214544196-60c757ea5f9a76ab3f844f12.png
Margaret Mitchell
user
meg
6486818b43ba3df95402d2dd
Jessica Newman
6486818b43ba3df95402d2de
Marie-Therese Png
6486818b43ba3df95402d2df
Andrew Strait
6486818b43ba3df95402d2e0
Apostol Vassilev
Evaluating the Social Impact of Generative AI Systems in Systems and
  Society
Generative AI systems across modalities, ranging from text (including code),
image, audio, and video, have broad social impacts, but there is no official
standard for means of evaluating those impacts or for which impacts should be
evaluated. In this paper, we present a guide that moves toward a standard
approach in evaluating a base generative AI system for any modality in two
overarching categories: what can be evaluated in a base system independent of
context and what can be evaluated in a societal context. Importantly, this
refers to base systems that have no predetermined application or deployment
context, including a model itself, as well as system components, such as
training data. Our framework for a base system defines seven categories of
social impact: bias, stereotypes, and representational harms; cultural values
and sensitive content; disparate performance; privacy and data protection;
financial costs; environmental costs; and data and content moderation labor
costs. Suggested methods for evaluation apply to listed generative modalities
and analyses of the limitations of existing evaluations serve as a starting
point for necessary investment in future evaluations. We offer five overarching
categories for what can be evaluated in a broader societal context, each with
its own subcategories: trustworthiness and autonomy; inequality,
marginalization, and violence; concentration of authority; labor and
creativity; and ecosystem and environment. Each subcategory includes
recommendations for mitigating harm.
A framework is provided to evaluate the social impacts of generative AI systems across various modalities, both in terms of individual system and broader societal contexts.