Drawing Conclusions from Draws: Rethinking Preference Semantics in
  Arena-Style LLM Evaluation
In arena-style evaluation of large language models (LLMs), two LLMs respond
to a user query, and the user chooses the winning response or deems the
"battle" a draw, resulting in an adjustment to the ratings of both models. The
prevailing approach for modeling these rating dynamics is to view battles as
two-player game matches, as in chess, and apply the Elo rating system and its
derivatives. In this paper, we critically examine this paradigm. Specifically,
we question whether a draw genuinely means that the two models are equal and
hence whether their ratings should be equalized. Instead, we conjecture that
draws are more indicative of query difficulty: if the query is too easy, then
both models are more likely to succeed equally. On three real-world arena
datasets, we show that ignoring rating updates for draws yields a 1-3% relative
increase in battle outcome prediction accuracy (which includes draws) for all
four rating systems studied. Further analyses suggest that draws occur more for
queries rated as very easy and those as highly objective, with risk ratios of
1.37 and 1.35, respectively. We recommend future rating systems to reconsider
existing draw semantics and to account for query properties in rating updates.
68dfda5f73e20ab5778419e0
Raphael Tang
claimed_verified
2025-10-05T12:46:30.840Z
63250bb8d206fe7b2d2f1b8b
https://cdn-avatars.huggingface.co/v1/production/uploads/1668053024087-63250bb8d206fe7b2d2f1b8b.jpeg
Raphael Tang
user
tetrisd
68dfda5f73e20ab5778419e1
Crystina Zhang
68dfda5f73e20ab5778419e2
Wenyan Li
68dfda5f73e20ab5778419e3
Carmen Lai
68dfda5f73e20ab5778419e4
Pontus Stenetorp
68dfda5f73e20ab5778419e5
Yao Lu
claimed_verified
2025-10-06T03:21:15.379Z
64119ace518736207a51625a
https://cdn-avatars.huggingface.co/v1/production/uploads/64119ace518736207a51625a/bJ-hyscgoU7a9gs-br0H7.jpeg
Yao Lu
user
yaolu
Drawing Conclusions from Draws: Rethinking Preference Semantics in
  Arena-Style LLM Evaluation
In arena-style evaluation of large language models (LLMs), two LLMs respond
to a user query, and the user chooses the winning response or deems the
"battle" a draw, resulting in an adjustment to the ratings of both models. The
prevailing approach for modeling these rating dynamics is to view battles as
two-player game matches, as in chess, and apply the Elo rating system and its
derivatives. In this paper, we critically examine this paradigm. Specifically,
we question whether a draw genuinely means that the two models are equal and
hence whether their ratings should be equalized. Instead, we conjecture that
draws are more indicative of query difficulty: if the query is too easy, then
both models are more likely to succeed equally. On three real-world arena
datasets, we show that ignoring rating updates for draws yields a 1-3% relative
increase in battle outcome prediction accuracy (which includes draws) for all
four rating systems studied. Further analyses suggest that draws occur more for
queries rated as very easy and those as highly objective, with risk ratios of
1.37 and 1.35, respectively. We recommend future rating systems to reconsider
existing draw semantics and to account for query properties in rating updates.
Ignoring rating updates for draws in arena-style evaluations of large language models improves battle outcome prediction accuracy by 1-3% across different rating systems.
arena-style evaluation large language models Elo rating system battle outcome prediction query difficulty query properties