CLASH: Evaluating Language Models on Judging High-Stakes Dilemmas from
  Multiple Perspectives
Navigating high-stakes dilemmas involving conflicting values is challenging
even for humans, let alone for AI. Yet prior work in evaluating the reasoning
capabilities of large language models (LLMs) in such situations has been
limited to everyday scenarios. To close this gap, this work first introduces
CLASH (Character perspective-based LLM Assessments in Situations with
High-stakes), a meticulously curated dataset consisting of 345 high-impact
dilemmas along with 3,795 individual perspectives of diverse values. In
particular, we design CLASH in a way to support the study of critical aspects
of value-based decision-making processes which are missing from prior work,
including understanding decision ambivalence and psychological discomfort as
well as capturing the temporal shifts of values in characters' perspectives. By
benchmarking 10 open and closed frontier models, we uncover several key
findings. (1) Even the strongest models, such as GPT-4o and Claude-Sonnet,
achieve less than 50% accuracy in identifying situations where the decision
should be ambivalent, while they perform significantly better in clear-cut
scenarios. (2) While LLMs reasonably predict psychological discomfort as marked
by human, they inadequately comprehend perspectives involving value shifts,
indicating a need for LLMs to reason over complex values. (3) Our experiments
also reveal a significant correlation between LLMs' value preferences and their
steerability towards a given value. (4) Finally, LLMs exhibit greater
steerability when engaged in value reasoning from a third-party perspective,
compared to a first-person setup, though certain value pairs benefit uniquely
from the first-person framing.
68003da4c771c307fec04da7
Ayoung Lee
claimed_verified
2025-04-20T15:03:37.028Z
66d079d94b3b38cefaf1dc4e
/avatars/f680a19fb7b8e52c811eb6df218a2cea.svg
Lee
user
Ayoung01
68003da4c771c307fec04da8
Ryan Sungmo Kwon
68003da4c771c307fec04da9
Peter Railton
68003da4c771c307fec04daa
Lu Wang
CLASH: Evaluating Language Models on Judging High-Stakes Dilemmas from
  Multiple Perspectives
Navigating high-stakes dilemmas involving conflicting values is challenging
even for humans, let alone for AI. Yet prior work in evaluating the reasoning
capabilities of large language models (LLMs) in such situations has been
limited to everyday scenarios. To close this gap, this work first introduces
CLASH (Character perspective-based LLM Assessments in Situations with
High-stakes), a meticulously curated dataset consisting of 345 high-impact
dilemmas along with 3,795 individual perspectives of diverse values. In
particular, we design CLASH in a way to support the study of critical aspects
of value-based decision-making processes which are missing from prior work,
including understanding decision ambivalence and psychological discomfort as
well as capturing the temporal shifts of values in characters' perspectives. By
benchmarking 10 open and closed frontier models, we uncover several key
findings. (1) Even the strongest models, such as GPT-4o and Claude-Sonnet,
achieve less than 50% accuracy in identifying situations where the decision
should be ambivalent, while they perform significantly better in clear-cut
scenarios. (2) While LLMs reasonably predict psychological discomfort as marked
by human, they inadequately comprehend perspectives involving value shifts,
indicating a need for LLMs to reason over complex values. (3) Our experiments
also reveal a significant correlation between LLMs' value preferences and their
steerability towards a given value. (4) Finally, LLMs exhibit greater
steerability when engaged in value reasoning from a third-party perspective,
compared to a first-person setup, though certain value pairs benefit uniquely
from the first-person framing.
LLMs perform well in clear-cut ethical dilemmas but struggle with ambiguity, psychological discomfort, and value shifts in high-stakes scenarios, showing differences in accuracy and steerability based on perspective.
large language models LLMs CLASH high-impact dilemmas decision ambivalence psychological discomfort value shifts value preferences steerability third-party perspective first-person perspective