Democratizing Diplomacy: A Harness for Evaluating Any Large Language
  Model on Full-Press Diplomacy
We present the first evaluation harness that enables any out-of-the-box,
local, Large Language Models (LLMs) to play full-press Diplomacy without
fine-tuning or specialized training. Previous work required frontier LLMs, or
fine-tuning, due to the high complexity and information density of Diplomacy's
game state. Combined with the high variance of matches, these factors made
Diplomacy prohibitive for study. In this work, we used data-driven iteration to
optimize a textual game state representation such that a 24B model can reliably
complete matches without any fine tuning. We develop tooling to facilitate
hypothesis testing and statistical analysis, and we present case studies on
persuasion, aggressive playstyles, and performance across a range of models. We
conduct a variety of experiments across many popular LLMs, finding the larger
models perform the best, but the smaller models still play adequately. We also
introduce Critical State Analysis: an experimental protocol for rapidly
iterating and analyzing key moments in a game at depth. Our harness
democratizes the evaluation of strategic reasoning in LLMs by eliminating the
need for fine-tuning, and it provides insights into how these capabilities
emerge naturally from widely used LLMs. Our code is available in the supplement
and will be open sourced.
689b421bfab6fdd2e52ac6d8
Alexander Duffy
claimed_verified
2025-08-13T07:14:22.861Z
689b418f0b10f771fb5c780a
/avatars/553a9d6866b7a3d94d9cb09e764ec32b.svg
Alex Duffy
user
Alex-GSL
689b421bfab6fdd2e52ac6d9
Samuel J Paech
689b421bfab6fdd2e52ac6da
Ishana Shastri
689b421bfab6fdd2e52ac6db
Elizabeth Karpinski
689b421bfab6fdd2e52ac6dc
Baptiste Alloui-Cros
689b421bfab6fdd2e52ac6dd
Tyler Marques
claimed_verified
2025-08-13T07:14:19.902Z
646535720e6c7618f6138daa
https://cdn-avatars.huggingface.co/v1/production/uploads/646535720e6c7618f6138daa/w1mFcDWodCtfwa6wB_EJZ.jpeg
Tyler Marques
user
tmarques
689b421bfab6fdd2e52ac6de
Matthew Lyle Olson
Democratizing Diplomacy: A Harness for Evaluating Any Large Language
  Model on Full-Press Diplomacy
We present the first evaluation harness that enables any out-of-the-box,
local, Large Language Models (LLMs) to play full-press Diplomacy without
fine-tuning or specialized training. Previous work required frontier LLMs, or
fine-tuning, due to the high complexity and information density of Diplomacy's
game state. Combined with the high variance of matches, these factors made
Diplomacy prohibitive for study. In this work, we used data-driven iteration to
optimize a textual game state representation such that a 24B model can reliably
complete matches without any fine tuning. We develop tooling to facilitate
hypothesis testing and statistical analysis, and we present case studies on
persuasion, aggressive playstyles, and performance across a range of models. We
conduct a variety of experiments across many popular LLMs, finding the larger
models perform the best, but the smaller models still play adequately. We also
introduce Critical State Analysis: an experimental protocol for rapidly
iterating and analyzing key moments in a game at depth. Our harness
democratizes the evaluation of strategic reasoning in LLMs by eliminating the
need for fine-tuning, and it provides insights into how these capabilities
emerge naturally from widely used LLMs. Our code is available in the supplement
and will be open sourced.
An evaluation harness allows large language models to play Diplomacy without fine-tuning, providing insights into their strategic reasoning capabilities.
Large Language Models LLMs Diplomacy textual game state representation hypothesis testing statistical analysis Critical State Analysis strategic reasoning