Beyond Human Judgment: A Bayesian Evaluation of LLMs' Moral Values
  Understanding
How do large language models understand moral dimensions compared to humans?
  This first large-scale Bayesian evaluation of market-leading language models
provides the answer. In contrast to prior work using deterministic ground truth
(majority or inclusion rules), we model annotator disagreements to capture both
aleatoric uncertainty (inherent human disagreement) and epistemic uncertainty
(model domain sensitivity). We evaluate top language models (Claude Sonnet 4,
DeepSeek-V3, Llama 4 Maverick) across 250K+ annotations from ~700 annotators on
100K+ texts spanning social media, news, and forums.
  Our GPU-optimized Bayesian framework processed 1M+ model queries, revealing
that AI models typically rank among the top 25\% of human annotators, achieving
much better-than-average balanced accuracy. Importantly, we find that AI
produces far fewer false negatives than humans, highlighting their more
sensitive moral detection capabilities.
68a57e0438160bfd39426a3d
Maciej Skorski
claimed_verified
2025-08-20T08:12:46.084Z
646344a2efb4e8550484863e
https://cdn-avatars.huggingface.co/v1/production/uploads/646344a2efb4e8550484863e/aHZTPDVEmy_K0i7fuAzdr.jpeg
Maciej Skorski
user
maciejskorski
68a57e0438160bfd39426a3e
Alina Landowska
Beyond Human Judgment: A Bayesian Evaluation of LLMs' Moral Values
  Understanding
How do large language models understand moral dimensions compared to humans?
  This first large-scale Bayesian evaluation of market-leading language models
provides the answer. In contrast to prior work using deterministic ground truth
(majority or inclusion rules), we model annotator disagreements to capture both
aleatoric uncertainty (inherent human disagreement) and epistemic uncertainty
(model domain sensitivity). We evaluate top language models (Claude Sonnet 4,
DeepSeek-V3, Llama 4 Maverick) across 250K+ annotations from ~700 annotators on
100K+ texts spanning social media, news, and forums.
  Our GPU-optimized Bayesian framework processed 1M+ model queries, revealing
that AI models typically rank among the top 25\% of human annotators, achieving
much better-than-average balanced accuracy. Importantly, we find that AI
produces far fewer false negatives than humans, highlighting their more
sensitive moral detection capabilities.
A Bayesian evaluation framework assesses large language models' moral understanding by modeling human annotator disagreements, showing AI models perform well with fewer false negatives.
Bayesian evaluation aleatoric uncertainty epistemic uncertainty model domain sensitivity GPU-optimized Bayesian framework balanced accuracy moral detection capabilities