Replacing Judges with Juries: Evaluating LLM Generations with a Panel of
  Diverse Models
As Large Language Models (LLMs) have become more advanced, they have outpaced
our abilities to accurately evaluate their quality. Not only is finding data to
adequately probe particular model properties difficult, but evaluating the
correctness of a model's freeform generation alone is a challenge. To address
this, many evaluations now rely on using LLMs themselves as judges to score the
quality of outputs from other LLMs. Evaluations most commonly use a single
large model like GPT4. While this method has grown in popularity, it is costly,
has been shown to introduce intramodel bias, and in this work, we find that
very large models are often unnecessary. We propose instead to evaluate models
using a Panel of LLm evaluators (PoLL). Across three distinct judge settings
and spanning six different datasets, we find that using a PoLL composed of a
larger number of smaller models outperforms a single large judge, exhibits less
intra-model bias due to its composition of disjoint model families, and does so
while being over seven times less expensive.
66305fa1cee11621a131e9a0
Pat Verga
admin_assigned
2024-04-30T07:30:04.908Z
6610180904bc9d80c02be453
/avatars/9b8bd16a862e5004e907e2af5c18e1db.svg
Pat Verga
user
pat-cohere
66305fa1cee11621a131e9a1
Sebastian Hofstatter
admin_assigned
2024-04-30T07:30:16.294Z
60506d8a5f211e59bdaff26b
https://cdn-avatars.huggingface.co/v1/production/uploads/1616085391299-60506d8a5f211e59bdaff26b.jpeg
Sebastian Hofstätter
user
sebastian-hofstaetter
66305fa1cee11621a131e9a2
Sophia Althammer
admin_assigned
2024-04-30T07:30:23.276Z
642c574c53e76b4c22842843
/avatars/09095a843c5f0dc8af998c893e41cd64.svg
Sophia Althammer
user
salthammer
66305fa1cee11621a131e9a3
Yixuan Su
admin_assigned
2024-04-30T07:34:32.933Z
6144c40da8f4f7baca305243
/avatars/1f9bbb70f859f34563fdd0a7de4cd254.svg
Yixuan Su
user
pangpang666
66305fa1cee11621a131e9a4
Aleksandra Piktus
admin_assigned
2024-04-30T07:34:40.337Z
5f5f4f093c67af20d9945a6c
https://cdn-avatars.huggingface.co/v1/production/uploads/1600081648206-noauth.jpeg
Aleksandra Piktus
user
ola13
66305fa1cee11621a131e9a5
Arkady Arkhangorodsky
admin_assigned
2024-04-30T07:35:03.090Z
65327ba4c7530aa27f690d32
/avatars/3baa31fbc9d1b1b2b7ead56ea42eea0b.svg
Arkady Arkhangorodsky
user
arkadyark-cohere
66305fa1cee11621a131e9a6
Minjie Xu
admin_assigned
2024-04-30T07:35:12.387Z
63a49568658851481f770bb6
https://cdn-avatars.huggingface.co/v1/production/uploads/1671730818150-63a49568658851481f770bb6.jpeg
Minjie Xu
user
chokkyvista
66305fa1cee11621a131e9a7
Naomi White
admin_assigned
2024-04-30T07:35:18.649Z
660e71ef076a625565bca191
/avatars/0c8df0ea42aad295fb77c2abeba852cc.svg
Naomi White
user
naomiwhite
66305fa1cee11621a131e9a8
Patrick Lewis
Replacing Judges with Juries: Evaluating LLM Generations with a Panel of
  Diverse Models
As Large Language Models (LLMs) have become more advanced, they have outpaced
our abilities to accurately evaluate their quality. Not only is finding data to
adequately probe particular model properties difficult, but evaluating the
correctness of a model's freeform generation alone is a challenge. To address
this, many evaluations now rely on using LLMs themselves as judges to score the
quality of outputs from other LLMs. Evaluations most commonly use a single
large model like GPT4. While this method has grown in popularity, it is costly,
has been shown to introduce intramodel bias, and in this work, we find that
very large models are often unnecessary. We propose instead to evaluate models
using a Panel of LLm evaluators (PoLL). Across three distinct judge settings
and spanning six different datasets, we find that using a PoLL composed of a
larger number of smaller models outperforms a single large judge, exhibits less
intra-model bias due to its composition of disjoint model families, and does so
while being over seven times less expensive.
Evaluating Large Language Models using a Panel of smaller evaluators outperforms single large evaluators with reduced bias and cost.
Large Language Models LLMs evaluators PoLL model properties freeform generation GPT4 intramodel bias disjoint model families