Role-Playing Evaluation for Large Language Models
Large Language Models (LLMs) demonstrate a notable capacity for adopting
personas and engaging in role-playing. However, evaluating this ability
presents significant challenges, as human assessments are resource-intensive
and automated evaluations can be biased. To address this, we introduce
Role-Playing Eval (RPEval), a novel benchmark designed to assess LLM
role-playing capabilities across four key dimensions: emotional understanding,
decision-making, moral alignment, and in-character consistency. This article
details the construction of RPEval and presents baseline evaluations. Our code
and dataset are available at https://github.com/yelboudouri/RPEval
683b58eb84fbd4b28d8d891e
Yassine El Boudouri
claimed_verified
2025-06-02T07:42:17.658Z
6469408ab2321e47d3294414
/avatars/05621d33f9f6337bc66e59d9e81d05ef.svg
Yassine El Boudouri
user
yelboudouri
683b58eb84fbd4b28d8d891f
Walter Nuninger
683b58eb84fbd4b28d8d8920
Julian Alvarez
683b58eb84fbd4b28d8d8921
Yvan Peter
Role-Playing Evaluation for Large Language Models
Large Language Models (LLMs) demonstrate a notable capacity for adopting
personas and engaging in role-playing. However, evaluating this ability
presents significant challenges, as human assessments are resource-intensive
and automated evaluations can be biased. To address this, we introduce
Role-Playing Eval (RPEval), a novel benchmark designed to assess LLM
role-playing capabilities across four key dimensions: emotional understanding,
decision-making, moral alignment, and in-character consistency. This article
details the construction of RPEval and presents baseline evaluations. Our code
and dataset are available at https://github.com/yelboudouri/RPEval
A benchmark called Role-Playing Eval assesses Large Language Models in role-playing across emotional understanding, decision-making, moral alignment, and in-character consistency.
Large Language Models Role-Playing Eval emotional understanding decision-making moral alignment in-character consistency