Sotopia-RL: Reward Design for Social Intelligence
Social intelligence has become a critical capability for large language
models (LLMs), enabling them to engage effectively in real-world social tasks
such as accommodation, persuasion, collaboration, and negotiation.
Reinforcement learning (RL) is a natural fit for training socially intelligent
agents because it allows models to learn sophisticated strategies directly
through social interactions. However, social interactions have two key
characteristics that set barriers for RL training: (1) partial observability,
where utterances have indirect and delayed effects that complicate credit
assignment, and (2) multi-dimensionality, where behaviors such as
rapport-building or knowledge-seeking contribute indirectly to goal
achievement. These characteristics make Markov decision process (MDP)-based RL
with single-dimensional episode-level rewards inefficient and unstable. To
address these challenges, we propose Sotopia-RL, a novel framework that refines
coarse episode-level feedback into utterance-level, multi-dimensional rewards.
Utterance-level credit assignment mitigates partial observability by
attributing outcomes to individual utterances, while multi-dimensional rewards
capture the full richness of social interactions and reduce reward hacking.
Experiments in Sotopia, an open-ended social learning environment, demonstrate
that Sotopia-RL achieves state-of-the-art social goal completion scores (7.17
on Sotopia-hard and 8.31 on Sotopia-full), significantly outperforming existing
approaches. Ablation studies confirm the necessity of both utterance-level
credit assignment and multi-dimensional reward design for RL training. Our
implementation is publicly available at:
https://github.com/sotopia-lab/sotopia-rl.
6894156a741a16f544fbced2
Haofei Yu
6894156a741a16f544fbced3
Zhengyang Qi
6894156a741a16f544fbced4
Yining Zhao
6894156a741a16f544fbced5
Kolby Nottingham
6894156a741a16f544fbced6
Keyang Xuan
6894156a741a16f544fbced7
Bodhisattwa Prasad Majumder
6894156a741a16f544fbced8
Hao Zhu
claimed_verified
2025-08-11T06:49:45.327Z
61aa376688c20eebf1e8deb3
/avatars/7c11dcb232c73547d7d87834be287822.svg
Hao Zhu
user
ProKil
6894156a741a16f544fbced9
Paul Pu Liang
6894156a741a16f544fbceda
Jiaxuan You
Sotopia-RL: Reward Design for Social Intelligence
Social intelligence has become a critical capability for large language
models (LLMs), enabling them to engage effectively in real-world social tasks
such as accommodation, persuasion, collaboration, and negotiation.
Reinforcement learning (RL) is a natural fit for training socially intelligent
agents because it allows models to learn sophisticated strategies directly
through social interactions. However, social interactions have two key
characteristics that set barriers for RL training: (1) partial observability,
where utterances have indirect and delayed effects that complicate credit
assignment, and (2) multi-dimensionality, where behaviors such as
rapport-building or knowledge-seeking contribute indirectly to goal
achievement. These characteristics make Markov decision process (MDP)-based RL
with single-dimensional episode-level rewards inefficient and unstable. To
address these challenges, we propose Sotopia-RL, a novel framework that refines
coarse episode-level feedback into utterance-level, multi-dimensional rewards.
Utterance-level credit assignment mitigates partial observability by
attributing outcomes to individual utterances, while multi-dimensional rewards
capture the full richness of social interactions and reduce reward hacking.
Experiments in Sotopia, an open-ended social learning environment, demonstrate
that Sotopia-RL achieves state-of-the-art social goal completion scores (7.17
on Sotopia-hard and 8.31 on Sotopia-full), significantly outperforming existing
approaches. Ablation studies confirm the necessity of both utterance-level
credit assignment and multi-dimensional reward design for RL training. Our
implementation is publicly available at:
https://github.com/sotopia-lab/sotopia-rl.
Sotopia-RL, a novel reinforcement learning framework, enhances social intelligence in large language models by refining feedback into utterance-level, multi-dimensional rewards, improving performance in social tasks.
reinforcement learning socially intelligent agents partial observability multi-dimensionality Markov decision process episode-level rewards utterance-level credit assignment multi-dimensional rewards reward hacking social goal completion scores Sotopia Sotopia-RL