JuStRank: Benchmarking LLM Judges for System Ranking
Given the rapid progress of generative AI, there is a pressing need to
systematically compare and choose between the numerous models and
configurations available. The scale and versatility of such evaluations make
the use of LLM-based judges a compelling solution for this challenge.
Crucially, this approach requires first to validate the quality of the LLM
judge itself. Previous work has focused on instance-based assessment of LLM
judges, where a judge is evaluated over a set of responses, or response pairs,
while being agnostic to their source systems. We argue that this setting
overlooks critical factors affecting system-level ranking, such as a judge's
positive or negative bias towards certain systems. To address this gap, we
conduct the first large-scale study of LLM judges as system rankers. System
scores are generated by aggregating judgment scores over multiple system
outputs, and the judge's quality is assessed by comparing the resulting system
ranking to a human-based ranking. Beyond overall judge assessment, our analysis
provides a fine-grained characterization of judge behavior, including their
decisiveness and bias.
675bdf1ba301c0203a58d39d
Ariel Gera
claimed_verified
2024-12-13T15:15:38.275Z
6415ebd2107962562e9a0712
/avatars/dfd2e6d42c6110bb64b6a73c96eeb3f3.svg
Ariel Gera
user
arielgera
675bdf1ba301c0203a58d39e
Odellia Boni
admin_assigned
2024-12-16T11:21:02.044Z
6742ef68f6646c6a4e0488e8
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/06egZhjicM7TJ4gXvW8yH.png
odellia boni
user
odelliab
675bdf1ba301c0203a58d39f
Yotam Perlitz
claimed_verified
2024-12-18T09:53:33.385Z
61e01a92a24b774dac1f32d7
/avatars/307aac0c6cfe1879f6a5e5e1d6c10413.svg
Yotam Perlitz
user
per
675bdf1ba301c0203a58d3a0
Roy Bar-Haim
admin_assigned
2024-12-16T11:20:52.572Z
666d8753e70e5838d9716446
/avatars/6390f9069372a72e79cd331bf043908b.svg
Roy Bar Haim
user
roybarhaim
675bdf1ba301c0203a58d3a1
Lilach Eden
admin_assigned
2024-12-16T11:20:47.110Z
6694bdae1a0138cdbfc2799b
/avatars/8c42646280c327253493edc149d16a56.svg
Lilach Eden
user
LilachE
675bdf1ba301c0203a58d3a2
Asaf Yehudai
claimed_verified
2024-12-16T09:42:06.387Z
638324f862badff43269e588
/avatars/907a39a9b44fc8b7f3fad35858b01fb7.svg
Asaf Yehudai
user
Asaf-Yehudai
JuStRank: Benchmarking LLM Judges for System Ranking
Given the rapid progress of generative AI, there is a pressing need to
systematically compare and choose between the numerous models and
configurations available. The scale and versatility of such evaluations make
the use of LLM-based judges a compelling solution for this challenge.
Crucially, this approach requires first to validate the quality of the LLM
judge itself. Previous work has focused on instance-based assessment of LLM
judges, where a judge is evaluated over a set of responses, or response pairs,
while being agnostic to their source systems. We argue that this setting
overlooks critical factors affecting system-level ranking, such as a judge's
positive or negative bias towards certain systems. To address this gap, we
conduct the first large-scale study of LLM judges as system rankers. System
scores are generated by aggregating judgment scores over multiple system
outputs, and the judge's quality is assessed by comparing the resulting system
ranking to a human-based ranking. Beyond overall judge assessment, our analysis
provides a fine-grained characterization of judge behavior, including their
decisiveness and bias.
A study evaluates LLM judges for system-level ranking in generative AI, assessing their quality, decisiveness, and bias through comparison with human-based rankings.
LLM-based judges system-level ranking instance-based assessment judge quality human-based ranking decisiveness bias