Benchmarking LLMs for Political Science: A United Nations Perspective
Large Language Models (LLMs) have achieved significant advances in natural
language processing, yet their potential for high-stake political
decision-making remains largely unexplored. This paper addresses the gap by
focusing on the application of LLMs to the United Nations (UN) decision-making
process, where the stakes are particularly high and political decisions can
have far-reaching consequences. We introduce a novel dataset comprising
publicly available UN Security Council (UNSC) records from 1994 to 2024,
including draft resolutions, voting records, and diplomatic speeches. Using
this dataset, we propose the United Nations Benchmark (UNBench), the first
comprehensive benchmark designed to evaluate LLMs across four interconnected
political science tasks: co-penholder judgment, representative voting
simulation, draft adoption prediction, and representative statement generation.
These tasks span the three stages of the UN decision-making process--drafting,
voting, and discussing--and aim to assess LLMs' ability to understand and
simulate political dynamics. Our experimental analysis demonstrates the
potential and challenges of applying LLMs in this domain, providing insights
into their strengths and limitations in political science. This work
contributes to the growing intersection of AI and political science, opening
new avenues for research and practical applications in global governance. The
UNBench Repository can be accessed at:
https://github.com/yueqingliang1/UNBench.
67bbecd2039a172a715f7b50
Yueqing Liang
claimed_verified
2025-02-24T09:07:13.108Z
650d3ed2e063bef6d9c46da7
/avatars/d16b3d0d403c1953780f6532e9cf27ad.svg
Yueqing Liang
user
yueqingliang
67bbecd2039a172a715f7b51
Liangwei Yang
67bbecd2039a172a715f7b52
Chen Wang
67bbecd2039a172a715f7b53
Congying Xia
67bbecd2039a172a715f7b54
Rui Meng
67bbecd2039a172a715f7b55
Xiongxiao Xu
67bbecd2039a172a715f7b56
Haoran Wang
67bbecd2039a172a715f7b57
Ali Payani
67bbecd2039a172a715f7b58
Kai Shu
Benchmarking LLMs for Political Science: A United Nations Perspective
Large Language Models (LLMs) have achieved significant advances in natural
language processing, yet their potential for high-stake political
decision-making remains largely unexplored. This paper addresses the gap by
focusing on the application of LLMs to the United Nations (UN) decision-making
process, where the stakes are particularly high and political decisions can
have far-reaching consequences. We introduce a novel dataset comprising
publicly available UN Security Council (UNSC) records from 1994 to 2024,
including draft resolutions, voting records, and diplomatic speeches. Using
this dataset, we propose the United Nations Benchmark (UNBench), the first
comprehensive benchmark designed to evaluate LLMs across four interconnected
political science tasks: co-penholder judgment, representative voting
simulation, draft adoption prediction, and representative statement generation.
These tasks span the three stages of the UN decision-making process--drafting,
voting, and discussing--and aim to assess LLMs' ability to understand and
simulate political dynamics. Our experimental analysis demonstrates the
potential and challenges of applying LLMs in this domain, providing insights
into their strengths and limitations in political science. This work
contributes to the growing intersection of AI and political science, opening
new avenues for research and practical applications in global governance. The
UNBench Repository can be accessed at:
https://github.com/yueqingliang1/UNBench.
The paper presents a benchmark for evaluating LLMs in UN decision-making by assessing their performance in tasks related to judgment, voting simulation, draft adoption, and statement generation.
Large Language Models (LLMs) United Nations United Nations Security Council (UNSC) United Nations Benchmark (UNBench) co-penholder judgment representative voting simulation draft adoption prediction representative statement generation political science global governance