WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in
  the Wild
We introduce WildBench, an automated evaluation framework designed to
benchmark large language models (LLMs) using challenging, real-world user
queries. WildBench consists of 1,024 tasks carefully selected from over one
million human-chatbot conversation logs. For automated evaluation with
WildBench, we have developed two metrics, WB-Reward and WB-Score, which are
computable using advanced LLMs such as GPT-4-turbo. WildBench evaluation uses
task-specific checklists to evaluate model outputs systematically and provides
structured explanations that justify the scores and comparisons, resulting in
more reliable and interpretable automatic judgments. WB-Reward employs
fine-grained pairwise comparisons between model responses, generating five
potential outcomes: much better, slightly better, slightly worse, much worse,
or a tie. Unlike previous evaluations that employed a single baseline model, we
selected three baseline models at varying performance levels to ensure a
comprehensive pairwise evaluation. Additionally, we propose a simple method to
mitigate length bias, by converting outcomes of ``slightly better/worse'' to
``tie'' if the winner response exceeds the loser one by more than K
characters. WB-Score evaluates the quality of model outputs individually,
making it a fast and cost-efficient evaluation metric. WildBench results
demonstrate a strong correlation with the human-voted Elo ratings from Chatbot
Arena on hard tasks. Specifically, WB-Reward achieves a Pearson correlation of
0.98 with top-ranking models. Additionally, WB-Score reaches 0.95, surpassing
both ArenaHard's 0.91 and AlpacaEval2.0's 0.89 for length-controlled win rates,
as well as the 0.87 for regular win rates.
66665de49f5a0b322905de64
Bill Yuchen Lin
admin_assigned
2024-06-10T10:19:45.095Z
607f666a4ad99100d63ce35c
https://cdn-avatars.huggingface.co/v1/production/uploads/607f666a4ad99100d63ce35c/QxhxnvfeV6efkxwUFHwjI.png
Bill Yuchen Lin
user
yuchenlin
66665de49f5a0b322905de65
Yuntian Deng
admin_assigned
2024-06-10T10:19:52.064Z
63081e15a670ed10f9d44229
https://cdn-avatars.huggingface.co/v1/production/uploads/63081e15a670ed10f9d44229/w1b9uq-9774bMMgJbSPsS.jpeg
Yuntian Deng
user
yuntian-deng
66665de49f5a0b322905de66
Khyathi Chandu
admin_assigned
2024-06-10T10:20:22.751Z
62ebe6472f8999993a847bb6
/avatars/29fb590f8116bdc9c55f7a333e2c4280.svg
Khyathi Raghavi Chandu
user
khyathi
66665de49f5a0b322905de67
Faeze Brahman
admin_assigned
2024-06-10T10:20:29.009Z
65282b8d578679aac7888aec
https://cdn-avatars.huggingface.co/v1/production/uploads/65282b8d578679aac7888aec/dibBkhH-z1c70mJZZxJ7u.jpeg
Faeze Brahman
user
faezeb
66665de49f5a0b322905de68
Abhilasha Ravichander
admin_assigned
2024-06-10T10:20:49.502Z
6349886c429608888c42319a
/avatars/f84b5fe8b76172878274754e3399d6ec.svg
Abhilasha Ravichander
user
lasha-nlp
66665de49f5a0b322905de69
Valentina Pyatkin
admin_assigned
2024-06-10T10:20:56.401Z
6556cff80e7a7067a934445f
https://cdn-avatars.huggingface.co/v1/production/uploads/6556cff80e7a7067a934445f/PoT7qQ6tVqLGGrYvGBbr8.jpeg
Valentina Pyatkin
user
valpy
66665de49f5a0b322905de6a
Nouha Dziri
admin_assigned
2024-06-10T10:21:19.482Z
625888c2c964680d85d70bff
/avatars/2ba91e452c5ea577ec9f1f71abc2fa49.svg
Nouha Dziri
user
nouhadziri
66665de49f5a0b322905de6b
Ronan Le Bras
admin_assigned
2024-06-10T10:21:26.252Z
635049104e753c9940fefd71
https://cdn-avatars.huggingface.co/v1/production/uploads/635049104e753c9940fefd71/HgR43XIFw3dneY5ufrAE8.jpeg
Ronan Le Bras
user
ronanlb
66665de49f5a0b322905de6c
Yejin Choi
admin_assigned
2024-06-10T10:21:33.126Z
64d42729f63b01b7f676b176
/avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
Yejin Choi
user
yejinchoinka
WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in
  the Wild
We introduce WildBench, an automated evaluation framework designed to
benchmark large language models (LLMs) using challenging, real-world user
queries. WildBench consists of 1,024 tasks carefully selected from over one
million human-chatbot conversation logs. For automated evaluation with
WildBench, we have developed two metrics, WB-Reward and WB-Score, which are
computable using advanced LLMs such as GPT-4-turbo. WildBench evaluation uses
task-specific checklists to evaluate model outputs systematically and provides
structured explanations that justify the scores and comparisons, resulting in
more reliable and interpretable automatic judgments. WB-Reward employs
fine-grained pairwise comparisons between model responses, generating five
potential outcomes: much better, slightly better, slightly worse, much worse,
or a tie. Unlike previous evaluations that employed a single baseline model, we
selected three baseline models at varying performance levels to ensure a
comprehensive pairwise evaluation. Additionally, we propose a simple method to
mitigate length bias, by converting outcomes of ``slightly better/worse'' to
``tie'' if the winner response exceeds the loser one by more than K
characters. WB-Score evaluates the quality of model outputs individually,
making it a fast and cost-efficient evaluation metric. WildBench results
demonstrate a strong correlation with the human-voted Elo ratings from Chatbot
Arena on hard tasks. Specifically, WB-Reward achieves a Pearson correlation of
0.98 with top-ranking models. Additionally, WB-Score reaches 0.95, surpassing
both ArenaHard's 0.91 and AlpacaEval2.0's 0.89 for length-controlled win rates,
as well as the 0.87 for regular win rates.
WildBench evaluates large language models using human-chatbot queries, providing tasks and metrics that correlate well with human-voted evaluations.
WildBench large language models LLMs user queries human-chatbot conversation logs WB-Reward WB-Score task-specific checklists fine-grained pairwise comparisons baseline models length bias Pearson correlation Elo ratings ArenaHard AlpacaEval2.0