Judging the Judges: A Collection of LLM-Generated Relevance Judgements
Using Large Language Models (LLMs) for relevance assessments offers promising
opportunities to improve Information Retrieval (IR), Natural Language
Processing (NLP), and related fields. Indeed, LLMs hold the promise of allowing
IR experimenters to build evaluation collections with a fraction of the manual
human labor currently required. This could help with fresh topics on which
there is still limited knowledge and could mitigate the challenges of
evaluating ranking systems in low-resource scenarios, where it is challenging
to find human annotators. Given the fast-paced recent developments in the
domain, many questions concerning LLMs as assessors are yet to be answered.
Among the aspects that require further investigation, we can list the impact of
various components in a relevance judgment generation pipeline, such as the
prompt used or the LLM chosen.
  This paper benchmarks and reports on the results of a large-scale automatic
relevance judgment evaluation, the LLMJudge challenge at SIGIR 2024, where
different relevance assessment approaches were proposed. In detail, we release
and benchmark 42 LLM-generated labels of the TREC 2023 Deep Learning track
relevance judgments produced by eight international teams who participated in
the challenge. Given their diverse nature, these automatically generated
relevance judgments can help the community not only investigate systematic
biases caused by LLMs but also explore the effectiveness of ensemble models,
analyze the trade-offs between different models and human assessors, and
advance methodologies for improving automated evaluation techniques. The
released resource is available at the following link:
https://llm4eval.github.io/LLMJudge-benchmark/
67b75ce1fedef65ff99cf5f8
Hossein A. Rahmani
claimed_verified
2025-02-20T16:57:36.417Z
64108fc514215c0775e13f5e
https://cdn-avatars.huggingface.co/v1/production/uploads/64108fc514215c0775e13f5e/pHWr8TlBnYrulo2owIrrv.jpeg
Hossein A. (Saeed) Rahmani
user
rahmanidashti
67b75ce1fedef65ff99cf5f9
Clemencia Siro
67b75ce1fedef65ff99cf5fa
Mohammad Aliannejadi
67b75ce1fedef65ff99cf5fb
Nick Craswell
67b75ce1fedef65ff99cf5fc
Charles L. A. Clarke
extracted_pending
2025-07-28T00:24:44.341Z
6886c302bd9f8b88ea84106d
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/ciRDKbtDsnYS-o5Cj_Nlh.png
Charles Clarke
user
claclark
67b75ce1fedef65ff99cf5fd
Guglielmo Faggioli
67b75ce1fedef65ff99cf5fe
Bhaskar Mitra
67b75ce1fedef65ff99cf5ff
Paul Thomas
67b75ce1fedef65ff99cf600
Emine Yilmaz
Judging the Judges: A Collection of LLM-Generated Relevance Judgements
Using Large Language Models (LLMs) for relevance assessments offers promising
opportunities to improve Information Retrieval (IR), Natural Language
Processing (NLP), and related fields. Indeed, LLMs hold the promise of allowing
IR experimenters to build evaluation collections with a fraction of the manual
human labor currently required. This could help with fresh topics on which
there is still limited knowledge and could mitigate the challenges of
evaluating ranking systems in low-resource scenarios, where it is challenging
to find human annotators. Given the fast-paced recent developments in the
domain, many questions concerning LLMs as assessors are yet to be answered.
Among the aspects that require further investigation, we can list the impact of
various components in a relevance judgment generation pipeline, such as the
prompt used or the LLM chosen.
  This paper benchmarks and reports on the results of a large-scale automatic
relevance judgment evaluation, the LLMJudge challenge at SIGIR 2024, where
different relevance assessment approaches were proposed. In detail, we release
and benchmark 42 LLM-generated labels of the TREC 2023 Deep Learning track
relevance judgments produced by eight international teams who participated in
the challenge. Given their diverse nature, these automatically generated
relevance judgments can help the community not only investigate systematic
biases caused by LLMs but also explore the effectiveness of ensemble models,
analyze the trade-offs between different models and human assessors, and
advance methodologies for improving automated evaluation techniques. The
released resource is available at the following link:
https://llm4eval.github.io/LLMJudge-benchmark/
The LLMJudge challenge at SIGIR 2024 benchmarks LLM-generated relevance judgments for the TREC 2023 Deep Learning track, providing insights into systematic biases, ensemble models, and trade-offs with human assessors in automated evaluation.
Large Language Models relevance assessments Information Retrieval Natural Language Processing Irrelevance judgment LLMJudge challenge TREC 2023 Deep Learning track automated evaluation techniques