GLEE: A Unified Framework and Benchmark for Language-based Economic
  Environments
Large Language Models (LLMs) show significant potential in economic and
strategic interactions, where communication via natural language is often
prevalent. This raises key questions: Do LLMs behave rationally? Can they mimic
human behavior? Do they tend to reach an efficient and fair outcome? What is
the role of natural language in the strategic interaction? How do
characteristics of the economic environment influence these dynamics? These
questions become crucial concerning the economic and societal implications of
integrating LLM-based agents into real-world data-driven systems, such as
online retail platforms and recommender systems. While the ML community has
been exploring the potential of LLMs in such multi-agent setups, varying
assumptions, design choices and evaluation criteria across studies make it
difficult to draw robust and meaningful conclusions. To address this, we
introduce a benchmark for standardizing research on two-player, sequential,
language-based games. Inspired by the economic literature, we define three base
families of games with consistent parameterization, degrees of freedom and
economic measures to evaluate agents' performance (self-gain), as well as the
game outcome (efficiency and fairness). We develop an open-source framework for
interaction simulation and analysis, and utilize it to collect a dataset of LLM
vs. LLM interactions across numerous game configurations and an additional
dataset of human vs. LLM interactions. Through extensive experimentation, we
demonstrate how our framework and dataset can be used to: (i) compare the
behavior of LLM-based agents to human players in various economic contexts;
(ii) evaluate agents in both individual and collective performance measures;
and (iii) quantify the effect of the economic characteristics of the
environments on the behavior of agents.
670686055dc3f2a798ad4a5f
Eilam Shapira
claimed_verified
2024-10-09T13:46:45.517Z
64802fb6c57f629056c59966
/avatars/d5ecabaceeba759969855acf512b6649.svg
Eilam Shapira
user
EilamSha
670686055dc3f2a798ad4a60
Omer Madmon
admin_assigned
2024-10-10T14:08:16.799Z
6707a02ae1e3c1a7b6f457b1
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/ZdwhsoEXaMPT1M29lz3uU.png
Omer Madmon
user
omermadmon
670686055dc3f2a798ad4a61
Itamar Reinman
admin_assigned
2024-10-10T09:22:28.588Z
64e67f74fdb84138027ae2b1
/avatars/f0bca499a30cfcfa058ac4bb31a08ee3.svg
Itamar Reinman
user
ireinman
670686055dc3f2a798ad4a62
Samuel Joseph Amouyal
670686055dc3f2a798ad4a63
Roi Reichart
670686055dc3f2a798ad4a64
Moshe Tennenholtz
GLEE: A Unified Framework and Benchmark for Language-based Economic
  Environments
Large Language Models (LLMs) show significant potential in economic and
strategic interactions, where communication via natural language is often
prevalent. This raises key questions: Do LLMs behave rationally? Can they mimic
human behavior? Do they tend to reach an efficient and fair outcome? What is
the role of natural language in the strategic interaction? How do
characteristics of the economic environment influence these dynamics? These
questions become crucial concerning the economic and societal implications of
integrating LLM-based agents into real-world data-driven systems, such as
online retail platforms and recommender systems. While the ML community has
been exploring the potential of LLMs in such multi-agent setups, varying
assumptions, design choices and evaluation criteria across studies make it
difficult to draw robust and meaningful conclusions. To address this, we
introduce a benchmark for standardizing research on two-player, sequential,
language-based games. Inspired by the economic literature, we define three base
families of games with consistent parameterization, degrees of freedom and
economic measures to evaluate agents' performance (self-gain), as well as the
game outcome (efficiency and fairness). We develop an open-source framework for
interaction simulation and analysis, and utilize it to collect a dataset of LLM
vs. LLM interactions across numerous game configurations and an additional
dataset of human vs. LLM interactions. Through extensive experimentation, we
demonstrate how our framework and dataset can be used to: (i) compare the
behavior of LLM-based agents to human players in various economic contexts;
(ii) evaluate agents in both individual and collective performance measures;
and (iii) quantify the effect of the economic characteristics of the
environments on the behavior of agents.
A benchmark for evaluating Large Language Models in strategic, language-based games provides insights into their rationality, mimicry of human behavior, and impact of economic characteristics on performance and outcome.
Large Language Models LLMs benchmark standardizing research two-player games sequential games language-based games economic measures interaction simulation dataset human vs. LLM interactions collective performance measures