Are they lovers or friends? Evaluating LLMs' Social Reasoning in English
  and Korean Dialogues
As large language models (LLMs) are increasingly used in human-AI
interactions, their social reasoning capabilities in interpersonal contexts are
critical. We introduce SCRIPTS, a 1k-dialogue dataset in English and Korean,
sourced from movie scripts. The task involves evaluating models' social
reasoning capability to infer the interpersonal relationships (e.g., friends,
sisters, lovers) between speakers in each dialogue. Each dialogue is annotated
with probabilistic relational labels (Highly Likely, Less Likely, Unlikely) by
native (or equivalent) Korean and English speakers from Korea and the U.S.
Evaluating nine models on our task, current proprietary LLMs achieve around
75-80% on the English dataset, whereas their performance on Korean drops to
58-69%. More strikingly, models select Unlikely relationships in 10-25% of
their responses. Furthermore, we find that thinking models and chain-of-thought
prompting, effective for general reasoning, provide minimal benefits for social
reasoning and occasionally amplify social biases. Our findings reveal
significant limitations in current LLMs' social reasoning capabilities,
highlighting the need for efforts to develop socially-aware language models.
68f9ad08b9b2e4ae046738d2
Eunsu Kim
claimed_verified
2025-10-23T14:46:34.645Z
6576ace7769f3ee9bd7b1b88
/avatars/5b5921e54413a37afde6ce017809c86e.svg
Eunsu Kim
user
EunsuKim
68f9ad08b9b2e4ae046738d3
Junyeong Park
68f9ad08b9b2e4ae046738d4
Juhyun Oh
68f9ad08b9b2e4ae046738d5
Kiwoong Park
68f9ad08b9b2e4ae046738d6
Seyoung Song
claimed_verified
2025-10-23T14:46:27.519Z
654f3cca8cc59d5b490b805b
https://cdn-avatars.huggingface.co/v1/production/uploads/654f3cca8cc59d5b490b805b/WAZZjN4q8VGp3VS2lAWyy.png
Seyoung Song
user
seyoungsong
68f9ad08b9b2e4ae046738d7
A. Seza Dogruoz
68f9ad08b9b2e4ae046738d8
Najoung Kim
68f9ad08b9b2e4ae046738d9
Alice Oh
Are they lovers or friends? Evaluating LLMs' Social Reasoning in English
  and Korean Dialogues
As large language models (LLMs) are increasingly used in human-AI
interactions, their social reasoning capabilities in interpersonal contexts are
critical. We introduce SCRIPTS, a 1k-dialogue dataset in English and Korean,
sourced from movie scripts. The task involves evaluating models' social
reasoning capability to infer the interpersonal relationships (e.g., friends,
sisters, lovers) between speakers in each dialogue. Each dialogue is annotated
with probabilistic relational labels (Highly Likely, Less Likely, Unlikely) by
native (or equivalent) Korean and English speakers from Korea and the U.S.
Evaluating nine models on our task, current proprietary LLMs achieve around
75-80% on the English dataset, whereas their performance on Korean drops to
58-69%. More strikingly, models select Unlikely relationships in 10-25% of
their responses. Furthermore, we find that thinking models and chain-of-thought
prompting, effective for general reasoning, provide minimal benefits for social
reasoning and occasionally amplify social biases. Our findings reveal
significant limitations in current LLMs' social reasoning capabilities,
highlighting the need for efforts to develop socially-aware language models.
Current large language models exhibit significant limitations in social reasoning, particularly in inferring interpersonal relationships across different languages, and thinking models or chain-of-thought prompting offer minimal improvement.
large language models social reasoning interpersonal relationships SCRIPTS dialogue dataset probabilistic relational labels thinking models chain-of-thought prompting socially-aware language models