Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in
  Large Language Models
Assessing how well a large language model (LLM) understands human, rather
than merely text, remains an open challenge. To bridge the gap, we introduce
Sentient Agent as a Judge (SAGE), an automated evaluation framework that
measures an LLM's higher-order social cognition. SAGE instantiates a Sentient
Agent that simulates human-like emotional changes and inner thoughts during
interaction, providing a more realistic evaluation of the tested model in
multi-turn conversations. At every turn, the agent reasons about (i) how its
emotion changes, (ii) how it feels, and (iii) how it should reply, yielding a
numerical emotion trajectory and interpretable inner thoughts. Experiments on
100 supportive-dialogue scenarios show that the final Sentient emotion score
correlates strongly with Barrett-Lennard Relationship Inventory (BLRI) ratings
and utterance-level empathy metrics, validating psychological fidelity. We also
build a public Sentient Leaderboard covering 18 commercial and open-source
models that uncovers substantial gaps (up to 4x) between frontier systems
(GPT-4o-Latest, Gemini2.5-Pro) and earlier baselines, gaps not reflected in
conventional leaderboards (e.g., Arena). SAGE thus provides a principled,
scalable and interpretable tool for tracking progress toward genuinely
empathetic and socially adept language agents.
681d7031e9969eecfcb4eb81
Bang Zhang
681d7031e9969eecfcb4eb82
Ruotian Ma
claimed_verified
2025-05-09T07:21:30.886Z
648294b2eb4befee378951c1
/avatars/da5d8bf9d8662cc2ffa2c0de49bd66a3.svg
Ruotian Ma
user
vvibt
681d7031e9969eecfcb4eb83
Qingxuan Jiang
681d7031e9969eecfcb4eb84
Peisong Wang
681d7031e9969eecfcb4eb85
Jiaqi Chen
681d7031e9969eecfcb4eb86
Zheng Xie
681d7031e9969eecfcb4eb87
Xingyu Chen
681d7031e9969eecfcb4eb88
Yue Wang
681d7031e9969eecfcb4eb89
Fanghua Ye
681d7031e9969eecfcb4eb8a
Jian Li
681d7031e9969eecfcb4eb8b
Yifan Yang
681d7031e9969eecfcb4eb8c
Zhaopeng Tu
claimed_verified
2025-05-09T07:21:28.468Z
67485743561b1e6f9579389f
/avatars/8a4cc63bd7be388010bc329bb74582a1.svg
Zhaopeng Tu
user
zptu
681d7031e9969eecfcb4eb8d
Xiaolong Li
Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in
  Large Language Models
Assessing how well a large language model (LLM) understands human, rather
than merely text, remains an open challenge. To bridge the gap, we introduce
Sentient Agent as a Judge (SAGE), an automated evaluation framework that
measures an LLM's higher-order social cognition. SAGE instantiates a Sentient
Agent that simulates human-like emotional changes and inner thoughts during
interaction, providing a more realistic evaluation of the tested model in
multi-turn conversations. At every turn, the agent reasons about (i) how its
emotion changes, (ii) how it feels, and (iii) how it should reply, yielding a
numerical emotion trajectory and interpretable inner thoughts. Experiments on
100 supportive-dialogue scenarios show that the final Sentient emotion score
correlates strongly with Barrett-Lennard Relationship Inventory (BLRI) ratings
and utterance-level empathy metrics, validating psychological fidelity. We also
build a public Sentient Leaderboard covering 18 commercial and open-source
models that uncovers substantial gaps (up to 4x) between frontier systems
(GPT-4o-Latest, Gemini2.5-Pro) and earlier baselines, gaps not reflected in
conventional leaderboards (e.g., Arena). SAGE thus provides a principled,
scalable and interpretable tool for tracking progress toward genuinely
empathetic and socially adept language agents.
SAGE, an automated evaluation framework, assesses the higher-order social cognition and empathy of large language models through simulated human-like emotions and inner thoughts in multi-turn conversations.
Sentient Agent as a Judge SAGE higher-order social cognition emotional changes inner thoughts numerical emotion trajectory Barrett-Lennard Relationship Inventory empathy metrics Sentient Leaderboard empathetic language agents