SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially?
Reasoning and strategic behavior in social interactions is a hallmark
of intelligence. This form of reasoning is significantly more sophisticated
than isolated planning or reasoning tasks in static settings (e.g., math
problem solving). In this paper, we present Strategic Planning,
Interaction, and Negotiation (SPIN-Bench), a new multi-domain
evaluation designed to measure the intelligence of strategic planning
and social reasoning. While many existing benchmarks focus on narrow
planning or single-agent reasoning, SPIN-Bench combines classical PDDL tasks,
competitive board games, cooperative card games, and multi-agent negotiation
scenarios in one unified framework. The framework includes both a benchmark as
well as an arena to simulate and evaluate the variety of social settings to
test reasoning and strategic behavior of AI agents. We formulate the benchmark
SPIN-Bench by systematically varying action spaces, state complexity, and the
number of interacting agents to simulate a variety of social settings where
success depends on not only methodical and step-wise decision making, but also
conceptual inference of other (adversarial or cooperative) participants.
Our experiments reveal that while contemporary LLMs handle basic fact
retrieval and short-range planning reasonably well, they encounter
significant performance bottlenecks in tasks requiring deep multi-hop
reasoning over large state spaces and socially adept coordination under
uncertainty. We envision SPIN-Bench as a catalyst for future research on robust
multi-agent planning, social reasoning, and human--AI teaming.
67d8e49df8b0e148f609c3a6
Jianzhu Yao
claimed_verified
2025-03-18T08:07:54.531Z
660dc90fab11c96e148f95f1
/avatars/b59fe4b7c16602ea33887a7517ebacb0.svg
Albert
user
yyyyyyjjjjzzz
67d8e49df8b0e148f609c3a7
Kevin Wang
claimed_verified
2025-03-18T14:58:01.093Z
66b979d9088299999ab714b7
/avatars/28f941da791e51788a4d8e2321c00ded.svg
Kevin Wang
user
Huanyu123
67d8e49df8b0e148f609c3a8
Ryan Hsieh
67d8e49df8b0e148f609c3a9
Haisu Zhou
claimed_verified
2025-03-18T08:07:43.752Z
672a86150957c590b21e49a2
/avatars/38b12f09af1ad994615b119cb3928404.svg
Monica Zhou
user
monicazhou1568
67d8e49df8b0e148f609c3aa
Tianqing Zou
67d8e49df8b0e148f609c3ab
Zerui Cheng
claimed_verified
2025-03-18T08:07:56.447Z
67d8e5f041fc44bb16e5468c
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/jUCe0rf7Z-cF51OC5Iczn.png
Zerui Cheng
user
ZeruiCheng
67d8e49df8b0e148f609c3ac
Zhangyang Wang
67d8e49df8b0e148f609c3ad
Pramod Viswanath
SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially?
Reasoning and strategic behavior in social interactions is a hallmark
of intelligence. This form of reasoning is significantly more sophisticated
than isolated planning or reasoning tasks in static settings (e.g., math
problem solving). In this paper, we present Strategic Planning,
Interaction, and Negotiation (SPIN-Bench), a new multi-domain
evaluation designed to measure the intelligence of strategic planning
and social reasoning. While many existing benchmarks focus on narrow
planning or single-agent reasoning, SPIN-Bench combines classical PDDL tasks,
competitive board games, cooperative card games, and multi-agent negotiation
scenarios in one unified framework. The framework includes both a benchmark as
well as an arena to simulate and evaluate the variety of social settings to
test reasoning and strategic behavior of AI agents. We formulate the benchmark
SPIN-Bench by systematically varying action spaces, state complexity, and the
number of interacting agents to simulate a variety of social settings where
success depends on not only methodical and step-wise decision making, but also
conceptual inference of other (adversarial or cooperative) participants.
Our experiments reveal that while contemporary LLMs handle basic fact
retrieval and short-range planning reasonably well, they encounter
significant performance bottlenecks in tasks requiring deep multi-hop
reasoning over large state spaces and socially adept coordination under
uncertainty. We envision SPIN-Bench as a catalyst for future research on robust
multi-agent planning, social reasoning, and human--AI teaming.
SPIN-Bench evaluates strategic planning and social reasoning in AI through a unified framework combining PDDL tasks, board games, card games, and negotiation scenarios, highlighting challenges in deep multi-hop reasoning and social coordination.
PDDL tasks competitive board games cooperative card games multi-agent negotiation action spaces state complexity interacting agents conceptual inference deep multi-hop reasoning socially adept coordination