Playing repeated games with Large Language Models
Large Language Models (LLMs) are transforming society and permeating into
diverse applications. As a result, LLMs will frequently interact with us and
other agents. It is, therefore, of great societal value to understand how LLMs
behave in interactive social settings. Here, we propose to use behavioral game
theory to study LLM's cooperation and coordination behavior. To do so, we let
different LLMs (GPT-3, GPT-3.5, and GPT-4) play finitely repeated games with
each other and with other, human-like strategies. Our results show that LLMs
generally perform well in such tasks and also uncover persistent behavioral
signatures. In a large set of two players-two strategies games, we find that
LLMs are particularly good at games where valuing their own self-interest pays
off, like the iterated Prisoner's Dilemma family. However, they behave
sub-optimally in games that require coordination. We, therefore, further focus
on two games from these distinct families. In the canonical iterated Prisoner's
Dilemma, we find that GPT-4 acts particularly unforgivingly, always defecting
after another agent has defected only once. In the Battle of the Sexes, we find
that GPT-4 cannot match the behavior of the simple convention to alternate
between options. We verify that these behavioral signatures are stable across
robustness checks. Finally, we show how GPT-4's behavior can be modified by
providing further information about the other player as well as by asking it to
predict the other player's actions before making a choice. These results enrich
our understanding of LLM's social behavior and pave the way for a behavioral
game theory for machines.
64740d8916ef9c21dbaa09ad
Elif Akata
admin_assigned
2023-05-30T13:21:58.670Z
6369176764aad59d4d3e9e9d
/avatars/1b501f1fb135de030d603e10c60a8b48.svg
Elif Akata
user
eakata
64740d8916ef9c21dbaa09ae
Lion Schulz
64740d8916ef9c21dbaa09af
Julian Coda-Forno
admin_assigned
2023-05-30T13:22:25.770Z
6475c4694a0bae093fe222df
/avatars/938c6268e732086905df724477625df8.svg
Julian Coda-Forno
user
Julian-CF
64740d8916ef9c21dbaa09b0
Seong Joon Oh
admin_assigned
2023-05-30T13:22:59.885Z
638a50450f10aa3064f03f23
/avatars/0c068458e42950c851758a238225c3a6.svg
Seong Joon Oh
user
coallaoh
64740d8916ef9c21dbaa09b1
Matthias Bethge
64740d8916ef9c21dbaa09b2
Eric Schulz
Playing repeated games with Large Language Models
Large Language Models (LLMs) are transforming society and permeating into
diverse applications. As a result, LLMs will frequently interact with us and
other agents. It is, therefore, of great societal value to understand how LLMs
behave in interactive social settings. Here, we propose to use behavioral game
theory to study LLM's cooperation and coordination behavior. To do so, we let
different LLMs (GPT-3, GPT-3.5, and GPT-4) play finitely repeated games with
each other and with other, human-like strategies. Our results show that LLMs
generally perform well in such tasks and also uncover persistent behavioral
signatures. In a large set of two players-two strategies games, we find that
LLMs are particularly good at games where valuing their own self-interest pays
off, like the iterated Prisoner's Dilemma family. However, they behave
sub-optimally in games that require coordination. We, therefore, further focus
on two games from these distinct families. In the canonical iterated Prisoner's
Dilemma, we find that GPT-4 acts particularly unforgivingly, always defecting
after another agent has defected only once. In the Battle of the Sexes, we find
that GPT-4 cannot match the behavior of the simple convention to alternate
between options. We verify that these behavioral signatures are stable across
robustness checks. Finally, we show how GPT-4's behavior can be modified by
providing further information about the other player as well as by asking it to
predict the other player's actions before making a choice. These results enrich
our understanding of LLM's social behavior and pave the way for a behavioral
game theory for machines.
LLMs exhibit varied behaviors in interactive games, performing well in self-interested games like the iterated Prisoner's Dilemma but sub-optimally in coordination games like the Battle of the Sexes, with behavior modifiable through additional information and prediction prompts.
Large Language Models (LLMs) behavioral game theory cooperation coordination finitely repeated games iterated Prisoner's Dilemma Battle of the Sexes