Debatable Intelligence: Benchmarking LLM Judges via Debate Speech
  Evaluation
We introduce Debate Speech Evaluation as a novel and challenging benchmark
for assessing LLM judges. Evaluating debate speeches requires a deep
understanding of the speech at multiple levels, including argument strength and
relevance, the coherence and organization of the speech, the appropriateness of
its style and tone, and so on. This task involves a unique set of cognitive
abilities that have previously received limited attention in systematic LLM
benchmarking. To explore such skills, we leverage a dataset of over 600
meticulously annotated debate speeches and present the first in-depth analysis
of how state-of-the-art LLMs compare to human judges on this task. Our findings
reveal a nuanced picture: while larger models can approximate individual human
judgments in some respects, they differ substantially in their overall judgment
behavior. We also investigate the ability of frontier LLMs to generate
persuasive, opinionated speeches, showing that models may perform at a human
level on this task.
684269c12be20aeb4aa81708
Noy Sternlicht
claimed_verified
2025-06-06T07:40:27.279Z
63e8914accae1fe5c6176a55
https://cdn-avatars.huggingface.co/v1/production/uploads/63e8914accae1fe5c6176a55/_gDjcqGKWKoSZrHSEIlAw.jpeg
Noy Sternlicht
user
noystl
684269c12be20aeb4aa81709
Ariel Gera
claimed_verified
2025-06-10T15:20:30.073Z
6415ebd2107962562e9a0712
/avatars/dfd2e6d42c6110bb64b6a73c96eeb3f3.svg
Ariel Gera
user
arielgera
684269c12be20aeb4aa8170a
Roy Bar-Haim
684269c12be20aeb4aa8170b
Tom Hope
684269c12be20aeb4aa8170c
Noam Slonim
Debatable Intelligence: Benchmarking LLM Judges via Debate Speech
  Evaluation
We introduce Debate Speech Evaluation as a novel and challenging benchmark
for assessing LLM judges. Evaluating debate speeches requires a deep
understanding of the speech at multiple levels, including argument strength and
relevance, the coherence and organization of the speech, the appropriateness of
its style and tone, and so on. This task involves a unique set of cognitive
abilities that have previously received limited attention in systematic LLM
benchmarking. To explore such skills, we leverage a dataset of over 600
meticulously annotated debate speeches and present the first in-depth analysis
of how state-of-the-art LLMs compare to human judges on this task. Our findings
reveal a nuanced picture: while larger models can approximate individual human
judgments in some respects, they differ substantially in their overall judgment
behavior. We also investigate the ability of frontier LLMs to generate
persuasive, opinionated speeches, showing that models may perform at a human
level on this task.
This study evaluates the performance of large language models in assessing debate speeches, a task requiring deep understanding of various aspects of speech, including argument strength and coherence, and compares it to human judges.
LLM large language models debate speech evaluation argument strength speech coherence opinionated speech generation