Training Language Models for Social Deduction with Multi-Agent
  Reinforcement Learning
Communicating in natural language is a powerful tool in multi-agent settings,
as it enables independent agents to share information in partially observable
settings and allows zero-shot coordination with humans. However, most prior
works are limited as they either rely on training with large amounts of human
demonstrations or lack the ability to generate natural and useful communication
strategies. In this work, we train language models to have productive
discussions about their environment in natural language without any human
demonstrations. We decompose the communication problem into listening and
speaking. Our key idea is to leverage the agent's goal to predict useful
information about the world as a dense reward signal that guides communication.
Specifically, we improve a model's listening skills by training them to predict
information about the environment based on discussions, and we simultaneously
improve a model's speaking skills with multi-agent reinforcement learning by
rewarding messages based on their influence on other agents. To investigate the
role and necessity of communication in complex social settings, we study an
embodied social deduction game based on Among Us, where the key question to
answer is the identity of an adversarial imposter. We analyze emergent
behaviors due to our technique, such as accusing suspects and providing
evidence, and find that it enables strong discussions, doubling the win rates
compared to standard RL. We release our code and models at
https://socialdeductionllm.github.io/
67ab1314385da1f07cda1271
Bidipta Sarkar
claimed_verified
2025-02-11T09:51:17.933Z
63abbf74ad514ca8d14a0548
/avatars/b1357b73b8f9a8ff9908710ad64154ef.svg
Bidipta Sarkar
user
bidiptas
67ab1314385da1f07cda1272
Warren Xia
67ab1314385da1f07cda1273
C. Karen Liu
67ab1314385da1f07cda1274
Dorsa Sadigh
Training Language Models for Social Deduction with Multi-Agent
  Reinforcement Learning
Communicating in natural language is a powerful tool in multi-agent settings,
as it enables independent agents to share information in partially observable
settings and allows zero-shot coordination with humans. However, most prior
works are limited as they either rely on training with large amounts of human
demonstrations or lack the ability to generate natural and useful communication
strategies. In this work, we train language models to have productive
discussions about their environment in natural language without any human
demonstrations. We decompose the communication problem into listening and
speaking. Our key idea is to leverage the agent's goal to predict useful
information about the world as a dense reward signal that guides communication.
Specifically, we improve a model's listening skills by training them to predict
information about the environment based on discussions, and we simultaneously
improve a model's speaking skills with multi-agent reinforcement learning by
rewarding messages based on their influence on other agents. To investigate the
role and necessity of communication in complex social settings, we study an
embodied social deduction game based on Among Us, where the key question to
answer is the identity of an adversarial imposter. We analyze emergent
behaviors due to our technique, such as accusing suspects and providing
evidence, and find that it enables strong discussions, doubling the win rates
compared to standard RL. We release our code and models at
https://socialdeductionllm.github.io/
Language models trained without human demonstrations improve multi-agent communication skills through reinforcement learning, enabling effective zero-shot coordination in complex social deduction tasks.
language models multi-agent reinforcement learning communication strategies dense reward signal embodiment social deduction adversarial imposter emergent behaviors zero-shot coordination