Investigating Human-Aligned Large Language Model Uncertainty
Recent work has sought to quantify large language model uncertainty to
facilitate model control and modulate user trust. Previous works focus on
measures of uncertainty that are theoretically grounded or reflect the average
overt behavior of the model. In this work, we investigate a variety of
uncertainty measures, in order to identify measures that correlate with human
group-level uncertainty. We find that Bayesian measures and a variation on
entropy measures, top-k entropy, tend to agree with human behavior as a
function of model size. We find that some strong measures decrease in
human-similarity with model size, but, by multiple linear regression, we find
that combining multiple uncertainty measures provide comparable human-alignment
with reduced size-dependency.
67d8f044f8b0e148f60cef0d
Kyle Moore
67d8f044f8b0e148f60cef0e
Jesse Roberts
extracted_pending
2025-03-18T04:02:14.260Z
63c19eb3a0ffa3857eae2efa
https://cdn-avatars.huggingface.co/v1/production/uploads/63c19eb3a0ffa3857eae2efa/pB-CbPa1QFTImiHanNCxU.jpeg
Jesse Roberts
user
JesseTNRoberts
67d8f044f8b0e148f60cef0f
Daryl Watson
67d8f044f8b0e148f60cef10
Pamela Wisniewski
Investigating Human-Aligned Large Language Model Uncertainty
Recent work has sought to quantify large language model uncertainty to
facilitate model control and modulate user trust. Previous works focus on
measures of uncertainty that are theoretically grounded or reflect the average
overt behavior of the model. In this work, we investigate a variety of
uncertainty measures, in order to identify measures that correlate with human
group-level uncertainty. We find that Bayesian measures and a variation on
entropy measures, top-k entropy, tend to agree with human behavior as a
function of model size. We find that some strong measures decrease in
human-similarity with model size, but, by multiple linear regression, we find
that combining multiple uncertainty measures provide comparable human-alignment
with reduced size-dependency.
Various uncertainty measures, particularly Bayesian and top-k entropy, are found to correlate with human uncertainty, and combining multiple measures can mitigate size-dependent human alignment issues.
Bayesian measures entropy measures top-k entropy multiple linear regression human-alignment size-dependency