Judging LLM-as-a-judge with MT-Bench and Chatbot Arena
Evaluating large language model (LLM) based chat assistants is challenging
due to their broad capabilities and the inadequacy of existing benchmarks in
measuring human preferences. To address this, we explore using strong LLMs as
judges to evaluate these models on more open-ended questions. We examine the
usage and limitations of LLM-as-a-judge, such as position and verbosity biases
and limited reasoning ability, and propose solutions to migrate some of them.
We then verify the agreement between LLM judges and human preferences by
introducing two benchmarks: MT-bench, a multi-turn question set; and Chatbot
Arena, a crowdsourced battle platform. Our results reveal that strong LLM
judges like GPT-4 can match both controlled and crowdsourced human preferences
well, achieving over 80\% agreement, the same level of agreement between
humans. Hence, LLM-as-a-judge is a scalable and explainable way to approximate
human preferences, which are otherwise very expensive to obtain. Additionally,
we show our benchmark and traditional benchmarks complement each other by
evaluating several variants of LLaMA/Vicuna. We will publicly release 80
MT-bench questions, 3K expert votes, and 30K conversations with human
preferences from Chatbot Arena.
64866c1360b30c3615c7d12a
Lianmin Zheng
claimed_verified
2023-06-12T18:59:04.900Z
62d35f3ceaf3858ce253ab7a
https://cdn-avatars.huggingface.co/v1/production/uploads/62d35f3ceaf3858ce253ab7a/3yiiMriltXJBi242FZY-W.jpeg
Lianmin
user
lmzheng
64866c1360b30c3615c7d12b
Wei-Lin Chiang
admin_assigned
2023-09-22T19:24:36.433Z
63071772fd79b417f1b92291
/avatars/ec68d20dcedf4553bb31d9f6e0ded813.svg
Wei-Lin Chiang
user
weichiang
64866c1360b30c3615c7d12c
Ying Sheng
admin_assigned
2023-06-13T13:46:25.519Z
6455b969654d8bccae50736c
https://cdn-avatars.huggingface.co/v1/production/uploads/6455b969654d8bccae50736c/WsK0_sbQSVeosi-fh-O_U.jpeg
Ying Sheng
user
ying1123
64866c1360b30c3615c7d12d
Siyuan Zhuang
64866c1360b30c3615c7d12e
Zhanghao Wu
admin_assigned
2023-06-13T13:46:01.420Z
6429b84e8852afdf89beeae5
/avatars/7cfc1a484077a8d511c46392de8dee07.svg
Zhanghao Wu
user
michaelvll
64866c1360b30c3615c7d12f
Yonghao Zhuang
64866c1360b30c3615c7d130
Zi Lin
64866c1360b30c3615c7d131
Zhuohan Li
64866c1360b30c3615c7d132
Dacheng Li
admin_assigned
2023-06-13T13:47:12.747Z
63715b25ffc0489ed7d1f415
https://cdn-avatars.huggingface.co/v1/production/uploads/63715b25ffc0489ed7d1f415/xZJepbs0LRqFbW1knnBKR.jpeg
Dacheng Li
user
DachengLi
64866c1360b30c3615c7d133
Eric. P Xing
64866c1360b30c3615c7d134
Hao Zhang
claimed_verified
2025-01-01T20:15:17.449Z
62d363143eebd640a4fa41fa
https://cdn-avatars.huggingface.co/v1/production/uploads/62d363143eebd640a4fa41fa/pvPwXlJ5OOb-UIfmffv4E.jpeg
Hao Zhang
user
zhisbug
64866c1360b30c3615c7d135
Joseph E. Gonzalez
64866c1360b30c3615c7d136
Ion Stoica
Judging LLM-as-a-judge with MT-Bench and Chatbot Arena
Evaluating large language model (LLM) based chat assistants is challenging
due to their broad capabilities and the inadequacy of existing benchmarks in
measuring human preferences. To address this, we explore using strong LLMs as
judges to evaluate these models on more open-ended questions. We examine the
usage and limitations of LLM-as-a-judge, such as position and verbosity biases
and limited reasoning ability, and propose solutions to migrate some of them.
We then verify the agreement between LLM judges and human preferences by
introducing two benchmarks: MT-bench, a multi-turn question set; and Chatbot
Arena, a crowdsourced battle platform. Our results reveal that strong LLM
judges like GPT-4 can match both controlled and crowdsourced human preferences
well, achieving over 80\% agreement, the same level of agreement between
humans. Hence, LLM-as-a-judge is a scalable and explainable way to approximate
human preferences, which are otherwise very expensive to obtain. Additionally,
we show our benchmark and traditional benchmarks complement each other by
evaluating several variants of LLaMA/Vicuna. We will publicly release 80
MT-bench questions, 3K expert votes, and 30K conversations with human
preferences from Chatbot Arena.
Using strong large language models as judges for evaluating other LLM-based chat assistants achieves high agreement with human preferences, offering a scalable and explainable solution compared to traditional benchmarks.
large language model LLM LLM-as-a-judge position bias verbosity bias reasoning ability MT-bench Chatbot Arena human preferences agreement scalability explainability LLaMA Vicuna controlled preferences crowdsourced preferences