A Framework for Automated Measurement of Responsible AI Harms in
  Generative AI Applications
We present a framework for the automated measurement of responsible AI (RAI)
metrics for large language models (LLMs) and associated products and services.
Our framework for automatically measuring harms from LLMs builds on existing
technical and sociotechnical expertise and leverages the capabilities of
state-of-the-art LLMs, such as GPT-4. We use this framework to run through
several case studies investigating how different LLMs may violate a range of
RAI-related principles. The framework may be employed alongside domain-specific
sociotechnical expertise to create measurements for new harm areas in the
future. By implementing this framework, we aim to enable more advanced harm
measurement efforts and further the responsible use of LLMs.
653f0e74e337eff8f51e3a89
Ahmed Magooda
653f0e74e337eff8f51e3a8a
Alec Helyar
admin_assigned
2023-10-30T13:57:08.521Z
6385710c20e00df40e4e5275
/avatars/bd793f0f456cbb1fca47ce9aca9c2acc.svg
Alec Helyar
user
alech97
653f0e74e337eff8f51e3a8b
Kyle Jackson
admin_assigned
2023-10-30T13:57:59.197Z
64a70f96fa4e39a4a653886c
/avatars/7e3fd4399cdf75bc9f01cf7b31db9fda.svg
Kyle Jackson
user
kjacks
653f0e74e337eff8f51e3a8c
David Sullivan
653f0e74e337eff8f51e3a8d
Chad Atalla
653f0e74e337eff8f51e3a8e
Emily Sheng
653f0e74e337eff8f51e3a8f
Dan Vann
653f0e74e337eff8f51e3a90
Richard Edgar
653f0e74e337eff8f51e3a91
Hamid Palangi
admin_assigned
2023-10-30T13:59:04.573Z
6189ddd0992df2640e3e7d40
/avatars/925f2308fc412ae352b57a1b71815028.svg
Hamid Palangi
user
hamidpalangi
653f0e74e337eff8f51e3a92
Roman Lutz
admin_assigned
2023-10-30T13:59:14.107Z
6429db4169bb3e94e9a28de7
/avatars/4e0ecef3f9f2fe0289de063e37d4dd68.svg
Roman Lutz
user
romanlutz
653f0e74e337eff8f51e3a93
Hongliang Kong
admin_assigned
2023-10-30T13:59:27.119Z
634ae8541fd4a53b672a3a77
/avatars/fff40c0e090dafc5e9ee566b04b1069d.svg
Kong
user
Hongliang
653f0e74e337eff8f51e3a94
Vincent Yun
653f0e74e337eff8f51e3a95
Eslam Kamal
admin_assigned
2023-10-30T13:59:45.625Z
63aa0841769a10efc400dc6b
/avatars/5add339a8006b2e827b8b2de03a60912.svg
ka
user
eslamkamal22
653f0e74e337eff8f51e3a96
Federico Zarfati
admin_assigned
2023-10-30T13:59:55.804Z
64c6d0a2568845d756442cb8
/avatars/90fade8b9cfc806ffb0595f5520cc62a.svg
Federico Zarfati
user
onitaku
653f0e74e337eff8f51e3a97
Hanna Wallach
653f0e74e337eff8f51e3a98
Sarah Bird
653f0e74e337eff8f51e3a99
Mei Chen
A Framework for Automated Measurement of Responsible AI Harms in
  Generative AI Applications
We present a framework for the automated measurement of responsible AI (RAI)
metrics for large language models (LLMs) and associated products and services.
Our framework for automatically measuring harms from LLMs builds on existing
technical and sociotechnical expertise and leverages the capabilities of
state-of-the-art LLMs, such as GPT-4. We use this framework to run through
several case studies investigating how different LLMs may violate a range of
RAI-related principles. The framework may be employed alongside domain-specific
sociotechnical expertise to create measurements for new harm areas in the
future. By implementing this framework, we aim to enable more advanced harm
measurement efforts and further the responsible use of LLMs.
A framework measures responsible AI metrics for large language models to assess and mitigate harm, leveraging existing expertise and advanced language model capabilities.
large language models LLMs responsible AI RAI GPT-4 sociotechnical expertise harm measurement