Are Large Language Model-based Evaluators the Solution to Scaling Up
  Multilingual Evaluation?
Large Language Models (LLMs) have demonstrated impressive performance on
Natural Language Processing (NLP) tasks, such as Question Answering,
Summarization, and Classification. The use of LLMs as evaluators, that can rank
or score the output of other models (usually LLMs) has become increasingly
popular, due to the limitations of current evaluation techniques including the
lack of appropriate benchmarks, metrics, cost, and access to human annotators.
While LLMs are capable of handling approximately 100 languages, the majority of
languages beyond the top 20 lack systematic evaluation across various tasks,
metrics, and benchmarks. This creates an urgent need to scale up multilingual
evaluation to ensure a precise understanding of LLM performance across diverse
languages. LLM-based evaluators seem like the perfect solution to this problem,
as they do not require human annotators, human-created references, or
benchmarks and can theoretically be used to evaluate any language covered by
the LLM. In this paper, we investigate whether LLM-based evaluators can help
scale up multilingual evaluation. Specifically, we calibrate LLM-based
evaluation against 20k human judgments of five metrics across three
text-generation tasks in eight languages. Our findings indicate that LLM-based
evaluators may exhibit bias towards higher scores and should be used with
caution and should always be calibrated with a dataset of native speaker
judgments, particularly in low-resource and non-Latin script languages.
6503bf26d9c245846b2f6a37
Rishav Hada
admin_assigned
2023-09-15T09:24:00.888Z
605229ff72705624f02496e2
/avatars/3a3353c8a509ddfc1096d1eb6d5ab576.svg
Rishav Hada
user
hadarishav
6503bf26d9c245846b2f6a38
Varun Gumma
admin_assigned
2023-09-15T09:24:15.868Z
64e5f244abf028d943a38350
/avatars/52ac934a28e354f5962e8a8b02ec887a.svg
Varun Gumma
user
VarunGumma
6503bf26d9c245846b2f6a39
Adrian de Wynter
admin_assigned
2023-09-15T09:24:29.206Z
61fb144d7884fbb4a3c5983c
https://cdn-avatars.huggingface.co/v1/production/uploads/1646244925841-61fb144d7884fbb4a3c5983c.jpeg
Adrian de Wynter
user
adewynter
6503bf26d9c245846b2f6a3a
Harshita Diddee
admin_assigned
2023-09-15T09:24:37.475Z
610e4e8370e452eb73ffbc87
/avatars/ea82b568f479f7df175da7f412383952.svg
Diddee
user
HarshitaDiddee
6503bf26d9c245846b2f6a3b
Mohamed Ahmed
6503bf26d9c245846b2f6a3c
Monojit Choudhury
6503bf26d9c245846b2f6a3d
Kalika Bali
6503bf26d9c245846b2f6a3e
Sunayana Sitaram
extracted_confirmed
2023-11-14T03:56:37.829Z
63edb3b7684767daecb04417
/avatars/9265619db1a1bd3c2470ef22e0d95dc4.svg
Sunayana Sitaram
user
Sunayana
Are Large Language Model-based Evaluators the Solution to Scaling Up
  Multilingual Evaluation?
Large Language Models (LLMs) have demonstrated impressive performance on
Natural Language Processing (NLP) tasks, such as Question Answering,
Summarization, and Classification. The use of LLMs as evaluators, that can rank
or score the output of other models (usually LLMs) has become increasingly
popular, due to the limitations of current evaluation techniques including the
lack of appropriate benchmarks, metrics, cost, and access to human annotators.
While LLMs are capable of handling approximately 100 languages, the majority of
languages beyond the top 20 lack systematic evaluation across various tasks,
metrics, and benchmarks. This creates an urgent need to scale up multilingual
evaluation to ensure a precise understanding of LLM performance across diverse
languages. LLM-based evaluators seem like the perfect solution to this problem,
as they do not require human annotators, human-created references, or
benchmarks and can theoretically be used to evaluate any language covered by
the LLM. In this paper, we investigate whether LLM-based evaluators can help
scale up multilingual evaluation. Specifically, we calibrate LLM-based
evaluation against 20k human judgments of five metrics across three
text-generation tasks in eight languages. Our findings indicate that LLM-based
evaluators may exhibit bias towards higher scores and should be used with
caution and should always be calibrated with a dataset of native speaker
judgments, particularly in low-resource and non-Latin script languages.
LLM-based evaluators show promise for scaling multilingual NLP tasks evaluation but may have biases toward higher scores and require calibration with native speaker judgments.
Large Language Models LLMs Natural Language Processing NLP Question Answering Summarization Classification multilingual evaluation human judgments text-generation tasks non-Latin script languages