Towards Measuring the Representation of Subjective Global Opinions in
  Language Models
Large language models (LLMs) may not equitably represent diverse global
perspectives on societal issues. In this paper, we develop a quantitative
framework to evaluate whose opinions model-generated responses are more similar
to. We first build a dataset, GlobalOpinionQA, comprised of questions and
answers from cross-national surveys designed to capture diverse opinions on
global issues across different countries. Next, we define a metric that
quantifies the similarity between LLM-generated survey responses and human
responses, conditioned on country. With our framework, we run three experiments
on an LLM trained to be helpful, honest, and harmless with Constitutional AI.
By default, LLM responses tend to be more similar to the opinions of certain
populations, such as those from the USA, and some European and South American
countries, highlighting the potential for biases. When we prompt the model to
consider a particular country's perspective, responses shift to be more similar
to the opinions of the prompted populations, but can reflect harmful cultural
stereotypes. When we translate GlobalOpinionQA questions to a target language,
the model's responses do not necessarily become the most similar to the
opinions of speakers of those languages. We release our dataset for others to
use and build on. Our data is at
https://huggingface.co/datasets/Anthropic/llm_global_opinions. We also provide
an interactive visualization at https://llmglobalvalues.anthropic.com.
649cd399dd87dd9ef76babcd
Esin Durmus
admin_assigned
2023-06-29T11:32:29.464Z
627dd7dbba19b19eff6aaa46
/avatars/6e43025ccb96e48339d1db477da043b0.svg
Esin Durmus
user
esdurmus
649cd399dd87dd9ef76babce
Karina Nyugen
649cd399dd87dd9ef76babcf
Thomas I. Liao
admin_assigned
2023-06-29T11:33:29.035Z
627c1e283974b0ed6b2d0f64
/avatars/9a828c6057ab4ea3d3cd8d939d12c2ae.svg
Thomas I. Liao
user
til
649cd399dd87dd9ef76babd0
Nicholas Schiefer
admin_assigned
2023-06-29T11:33:56.706Z
63a246e2931d2c13ad842a59
/avatars/9ac4f3d01c293ce97507be402dfa688c.svg
Nicholas Schiefer
user
nschiefer
649cd399dd87dd9ef76babd1
Amanda Askell
649cd399dd87dd9ef76babd2
Anton Bakhtin
649cd399dd87dd9ef76babd3
Carol Chen
649cd399dd87dd9ef76babd4
Zac Hatfield-Dodds
649cd399dd87dd9ef76babd5
Danny Hernandez
649cd399dd87dd9ef76babd6
Nicholas Joseph
649cd399dd87dd9ef76babd7
Liane Lovitt
649cd399dd87dd9ef76babd8
Sam McCandlish
649cd399dd87dd9ef76babd9
Orowa Sikder
admin_assigned
2023-06-29T11:36:14.959Z
62eab58af602390a049ae5a5
/avatars/73328c7ac59d072756ebad7c0548789b.svg
Orowa Sikder
user
Orowa
649cd399dd87dd9ef76babda
Alex Tamkin
admin_assigned
2023-06-29T11:36:54.419Z
5fc18c42ea82dd667bb0ffaf
/avatars/ae98f82442b560de247d2354cc364398.svg
Alex Tamkin
user
alextamkin
649cd399dd87dd9ef76babdb
Janel Thamkul
649cd399dd87dd9ef76babdc
Jared Kaplan
649cd399dd87dd9ef76babdd
Jack Clark
649cd399dd87dd9ef76babde
Deep Ganguli
admin_assigned
2023-06-29T11:38:13.949Z
63924e6b2bfa48413aa0a4ad
/avatars/f8219c475af6995b6fb9ebe727c5be43.svg
Deep Ganguli
user
dganguli
Towards Measuring the Representation of Subjective Global Opinions in
  Language Models
Large language models (LLMs) may not equitably represent diverse global
perspectives on societal issues. In this paper, we develop a quantitative
framework to evaluate whose opinions model-generated responses are more similar
to. We first build a dataset, GlobalOpinionQA, comprised of questions and
answers from cross-national surveys designed to capture diverse opinions on
global issues across different countries. Next, we define a metric that
quantifies the similarity between LLM-generated survey responses and human
responses, conditioned on country. With our framework, we run three experiments
on an LLM trained to be helpful, honest, and harmless with Constitutional AI.
By default, LLM responses tend to be more similar to the opinions of certain
populations, such as those from the USA, and some European and South American
countries, highlighting the potential for biases. When we prompt the model to
consider a particular country's perspective, responses shift to be more similar
to the opinions of the prompted populations, but can reflect harmful cultural
stereotypes. When we translate GlobalOpinionQA questions to a target language,
the model's responses do not necessarily become the most similar to the
opinions of speakers of those languages. We release our dataset for others to
use and build on. Our data is at
https://huggingface.co/datasets/Anthropic/llm_global_opinions. We also provide
an interactive visualization at https://llmglobalvalues.anthropic.com.
A quantitative framework evaluates the similarities between large language model responses and human perspectives on global issues, highlighting biases and limitations.
large language models GlobalOpinionQA cross-national surveys metric Constitutional AI country perspectives cultural stereotypes interactive visualization