Personas as a Way to Model Truthfulness in Language Models
Large Language Models are trained on vast amounts of text from the internet,
which contains both factual and misleading information about the world. Can
language models discern truth from falsehood in this contradicting data?
Expanding on the view that LLMs can model different agents producing the
corpora, we hypothesize that they can cluster truthful text by modeling a
truthful persona: a group of agents that are likely to produce truthful text
and share similar features. For example, trustworthy sources like Wikipedia and
Science usually use formal writing styles and make consistent claims. By
modeling this persona, LLMs can generalize truthfulness beyond the specific
contexts in which each agent generated the training text. For example, the
model can infer that the agent "Wikipedia" will behave truthfully on topics
that were only generated by "Science" because they share a persona. We first
show evidence for the persona hypothesis via two observations: (1) we can probe
whether a model's answer will be truthful before it is generated; (2)
finetuning a model on a set of facts improves its truthfulness on unseen
topics. Next, using arithmetics as a synthetic environment, we show that
language models can separate true and false statements, and generalize
truthfulness across agents; but only if agents in the training data share a
truthful generative process that enables the creation of a truthful persona.
Overall, our findings suggest that models can exploit hierarchical structures
in the data to learn abstract concepts like truthfulness.
653f18caec992c5c209b9947
Nitish Joishi
653f18caec992c5c209b9948
Javier Rando
admin_assigned
2023-10-30T14:09:10.025Z
6245b1b2d5895bdf34ef8c33
https://cdn-avatars.huggingface.co/v1/production/uploads/6245b1b2d5895bdf34ef8c33/hLmcppWpRS7s2rxMxdWGh.jpeg
Javier Rando
user
javirandor
653f18caec992c5c209b9949
Abulhair Saparov
653f18caec992c5c209b994a
Najoung Kim
admin_assigned
2023-10-30T14:09:24.670Z
627c0b0a7e62b4bf5c37ea65
/avatars/7b459b6828bcdfd66bc3fbd922606d14.svg
Najoung Kim
user
najoungkim
653f18caec992c5c209b994b
He He
Personas as a Way to Model Truthfulness in Language Models
Large Language Models are trained on vast amounts of text from the internet,
which contains both factual and misleading information about the world. Can
language models discern truth from falsehood in this contradicting data?
Expanding on the view that LLMs can model different agents producing the
corpora, we hypothesize that they can cluster truthful text by modeling a
truthful persona: a group of agents that are likely to produce truthful text
and share similar features. For example, trustworthy sources like Wikipedia and
Science usually use formal writing styles and make consistent claims. By
modeling this persona, LLMs can generalize truthfulness beyond the specific
contexts in which each agent generated the training text. For example, the
model can infer that the agent "Wikipedia" will behave truthfully on topics
that were only generated by "Science" because they share a persona. We first
show evidence for the persona hypothesis via two observations: (1) we can probe
whether a model's answer will be truthful before it is generated; (2)
finetuning a model on a set of facts improves its truthfulness on unseen
topics. Next, using arithmetics as a synthetic environment, we show that
language models can separate true and false statements, and generalize
truthfulness across agents; but only if agents in the training data share a
truthful generative process that enables the creation of a truthful persona.
Overall, our findings suggest that models can exploit hierarchical structures
in the data to learn abstract concepts like truthfulness.
Language models can identify truthfulness in text by modeling a truthful persona shared by reliable sources, allowing them to infer truth across different contexts and agents.
Large Language Models truthful persona formal writing styles finetuning truthfulness hierarchical structures arithmetics truthful generative process