Validating LLM-as-a-Judge Systems under Rating Task Indeterminacy
Luke Guerdan, Solon Barocas, Kenneth Holstein, Hanna Wallach, Steven Wu, Alex Chouldechova
The LLM-as-a-judge paradigm, in which a judge LLM system replaces human raters in rating the outputs of other generative AI (GenAI) systems, plays a critical role in scaling and standardizing GenAI evaluations. To validate such judge systems, evaluators assess human--judge agreement by first collecting multiple human ratings for each item in a validation corpus, then aggregating the ratings into a single, per-item gold label rating. In many cases, however, items or rating criteria may be vague or ambiguous, or admit multiple interpretations, so a given rater (human or LLM) may deem multiple answers "reasonable" or "correct."  We call such rating tasks indeterminate. Problematically, many indeterminate rating tasks rely on forced-choice elicitation, whereby raters are able to select only one answer option for each item.  In this paper, we introduce a framework for LLM-as-a-judge validation for indeterminate rating tasks.  We draw theoretical connections between different measures of judge system performance under different human--judge agreement metrics, and different rating elicitation and aggregation schemes.  We demonstrate that differences in how humans and LLMs resolve indeterminacy in forced-choice rating tasks heavily bias LLM-as-a-judge validations.    Through extensive experiments involving 11 real-world rating tasks and 8 commercial LLMs, we show that standard validation approaches select judge systems that are highly suboptimal, performing as much as 30\% worse than judge systems selected by alternatives we propose.  We conclude with concrete recommendations for more principled and reliable approaches to LLM-as-a-judge validation.