When Judgment Becomes Noise: How Design Failures in LLM Judge Benchmarks
  Silently Undermine Validity
LLM-judged benchmarks are increasingly used to evaluate complex model
behaviors, yet their design introduces failure modes absent in conventional
ground-truth based benchmarks. We argue that without tight objectives and
verifiable constructions, benchmark rankings can produce high-confidence
rankings that are in fact largely noise. We introduce two mechanisms to
diagnose these issues. Schematic adherence quantifies how much of a judge's
overall verdict is explained by the explicit evaluation schema, revealing
unexplained variance when judges deviate from their own rubric. Psychometric
validity aggregates internal consistency and discriminant validity signals to
quantify irreducible uncertainty in any benchmarking run. Applying these tools
to Arena-Hard Auto, we find severe schema incoherence and factor collapse
across popular judges: for example, unexplained variance exceeding 90 percent
for DeepSeek-R1-32B and factor correlations above 0.93 for most criteria. We
also show that the ELO-style aggregation used by Arena-Hard Auto collapses and
masks genuine ranking uncertainty. Our results highlight design failures that
undermine validity and offer actionable principles for building better-scoped,
reliability-aware LLM-judged benchmarks. We release our code at
https://anonymous.4open.science/r/judgment-to-noise-947D/README.md
68d66ad88ccd91bdd39fff17
Benjamin Feuer
admin_assigned
2025-09-26T16:16:29.294Z
62f7f4efe7c1c9bf10c81465
https://cdn-avatars.huggingface.co/v1/production/uploads/62f7f4efe7c1c9bf10c81465/AYlOg0fkP1o4GAP-8Y3xt.jpeg
Benjamin Feuer
user
penfever
68d66ad88ccd91bdd39fff18
Chiung-Yi Tseng
admin_assigned
2025-09-26T16:16:37.459Z
64f2a7dc64ab878b8a4d7e36
/avatars/cabd564f04a8fbf6a873e1f9d687daf1.svg
Chiung-Yi Tseng
user
chiungyi
68d66ad88ccd91bdd39fff19
Astitwa Sarthak Lathe
68d66ad88ccd91bdd39fff1a
Oussama Elachqar
admin_assigned
2025-09-26T16:16:47.018Z
64c88ce4f723215b6dc21be1
https://cdn-avatars.huggingface.co/v1/production/uploads/64c88ce4f723215b6dc21be1/m8BzBVYKYau2DmtMwpCGm.jpeg
Oussama  Elachqar
user
oelachqar
68d66ad88ccd91bdd39fff1b
John P Dickerson
admin_assigned
2025-09-26T16:17:26.285Z
631442aa8d85ad332f9e1145
/avatars/f176b605a918dcf5c11a289a605391e2.svg
John Dickerson
user
jdickerson
When Judgment Becomes Noise: How Design Failures in LLM Judge Benchmarks
  Silently Undermine Validity
LLM-judged benchmarks are increasingly used to evaluate complex model
behaviors, yet their design introduces failure modes absent in conventional
ground-truth based benchmarks. We argue that without tight objectives and
verifiable constructions, benchmark rankings can produce high-confidence
rankings that are in fact largely noise. We introduce two mechanisms to
diagnose these issues. Schematic adherence quantifies how much of a judge's
overall verdict is explained by the explicit evaluation schema, revealing
unexplained variance when judges deviate from their own rubric. Psychometric
validity aggregates internal consistency and discriminant validity signals to
quantify irreducible uncertainty in any benchmarking run. Applying these tools
to Arena-Hard Auto, we find severe schema incoherence and factor collapse
across popular judges: for example, unexplained variance exceeding 90 percent
for DeepSeek-R1-32B and factor correlations above 0.93 for most criteria. We
also show that the ELO-style aggregation used by Arena-Hard Auto collapses and
masks genuine ranking uncertainty. Our results highlight design failures that
undermine validity and offer actionable principles for building better-scoped,
reliability-aware LLM-judged benchmarks. We release our code at
https://anonymous.4open.science/r/judgment-to-noise-947D/README.md
LLM-judged benchmarks can produce unreliable rankings due to schema incoherence and factor collapse, which are diagnosed using schematic adherence and psychometric validity.
schematic adherence psychometric validity internal consistency discriminant validity schema incoherence factor collapse ELO-style aggregation