Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in
  Large Language Models
Assessing how well a large language model (LLM) understands human, rather
than merely text, remains an open challenge. To bridge the gap, we introduce
Sentient Agent as a Judge (SAGE), an automated evaluation framework that
measures an LLM's higher-order social cognition. SAGE instantiates a Sentient
Agent that simulates human-like emotional changes and inner thoughts during
interaction, providing a more realistic evaluation of the tested model in
multi-turn conversations. At every turn, the agent reasons about (i) how its
emotion changes, (ii) how it feels, and (iii) how it should reply, yielding a
numerical emotion trajectory and interpretable inner thoughts. Experiments on
100 supportive-dialogue scenarios show that the final Sentient emotion score
correlates strongly with Barrett-Lennard Relationship Inventory (BLRI) ratings
and utterance-level empathy metrics, validating psychological fidelity. We also
build a public Sentient Leaderboard covering 18 commercial and open-source
models that uncovers substantial gaps (up to 4x) between frontier systems
(GPT-4o-Latest, Gemini2.5-Pro) and earlier baselines, gaps not reflected in
conventional leaderboards (e.g., Arena). SAGE thus provides a principled,
scalable and interpretable tool for tracking progress toward genuinely
empathetic and socially adept language agents.
681d7031e9969eecfcb4eb81
Bang Zhang
681d7031e9969eecfcb4eb82
Ruotian Ma
claimed_verified
2025-05-09T07:21:30.886Z
648294b2eb4befee378951c1
/avatars/da5d8bf9d8662cc2ffa2c0de49bd66a3.svg
Ruotian Ma
user
vvibt
681d7031e9969eecfcb4eb83
Qingxuan Jiang
681d7031e9969eecfcb4eb84
Peisong Wang
claimed_verified
2025-07-09T08:54:06.149Z
64be4408c05a0df0d2b6012e
/avatars/09d8427505a418090391dc5a3f8bfef2.svg
PSWang
user
CedarWang
681d7031e9969eecfcb4eb85
Jiaqi Chen
681d7031e9969eecfcb4eb86
Zheng Xie
681d7031e9969eecfcb4eb87
Xingyu Chen
681d7031e9969eecfcb4eb88
Yue Wang
681d7031e9969eecfcb4eb89
Fanghua Ye
681d7031e9969eecfcb4eb8a
Jian Li
681d7031e9969eecfcb4eb8b
Yifan Yang
681d7031e9969eecfcb4eb8c
Zhaopeng Tu
claimed_verified
2025-05-09T07:21:28.468Z
67485743561b1e6f9579389f
/avatars/8a4cc63bd7be388010bc329bb74582a1.svg
Zhaopeng Tu
user
zptu
681d7031e9969eecfcb4eb8d
Xiaolong Li
Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in
  Large Language Models
Assessing how well a large language model (LLM) understands human, rather
than merely text, remains an open challenge. To bridge the gap, we introduce
Sentient Agent as a Judge (SAGE), an automated evaluation framework that
measures an LLM's higher-order social cognition. SAGE instantiates a Sentient
Agent that simulates human-like emotional changes and inner thoughts during
interaction, providing a more realistic evaluation of the tested model in
multi-turn conversations. At every turn, the agent reasons about (i) how its
emotion changes, (ii) how it feels, and (iii) how it should reply, yielding a
numerical emotion trajectory and interpretable inner thoughts. Experiments on
100 supportive-dialogue scenarios show that the final Sentient emotion score
correlates strongly with Barrett-Lennard Relationship Inventory (BLRI) ratings
and utterance-level empathy metrics, validating psychological fidelity. We also
build a public Sentient Leaderboard covering 18 commercial and open-source
models that uncovers substantial gaps (up to 4x) between frontier systems
(GPT-4o-Latest, Gemini2.5-Pro) and earlier baselines, gaps not reflected in
conventional leaderboards (e.g., Arena). SAGE thus provides a principled,
scalable and interpretable tool for tracking progress toward genuinely
empathetic and socially adept language agents.
SAGE, an automated evaluation framework, assesses the higher-order social cognition and empathy of large language models through simulated human-like emotions and inner thoughts in multi-turn conversations.
Sentient Agent as a Judge SAGE higher-order social cognition emotional changes inner thoughts numerical emotion trajectory Barrett-Lennard Relationship Inventory empathy metrics Sentient Leaderboard empathetic language agents