The Devil is in the Errors: Leveraging Large Language Models for
  Fine-grained Machine Translation Evaluation
Automatic evaluation of machine translation (MT) is a critical tool driving
the rapid iterative development of MT systems. While considerable progress has
been made on estimating a single scalar quality score, current metrics lack the
informativeness of more detailed schemes that annotate individual errors, such
as Multidimensional Quality Metrics (MQM). In this paper, we help fill this gap
by proposing AutoMQM, a prompting technique which leverages the reasoning and
in-context learning capabilities of large language models (LLMs) and asks them
to identify and categorize errors in translations. We start by evaluating
recent LLMs, such as PaLM and PaLM-2, through simple score prediction
prompting, and we study the impact of labeled data through in-context learning
and finetuning. We then evaluate AutoMQM with PaLM-2 models, and we find that
it improves performance compared to just prompting for scores (with
particularly large gains for larger models) while providing interpretability
through error spans that align with human annotations.
64db1421322a5774e09c01e2
Patrick Fernandes
admin_assigned
2023-08-15T13:19:11.910Z
6354206b12edd0ed5dc235f4
/avatars/c2f5872eb4cad979635293c0af47db03.svg
Patrick Fernandes
user
patricksf
64db1421322a5774e09c01e3
Daniel Deutsch
admin_assigned
2023-08-15T13:19:28.461Z
64d41bd5919f6f802e738af7
/avatars/916086b0e2850054c56a202afe46aa2f.svg
Daniel Deutsch
user
danieldeutsch
64db1421322a5774e09c01e4
Mara Finkelstein
64db1421322a5774e09c01e5
Parker Riley
64db1421322a5774e09c01e6
André F. T. Martins
64db1421322a5774e09c01e7
Graham Neubig
admin_assigned
2023-08-15T13:21:38.285Z
60de14638bedd2315529d43f
https://cdn-avatars.huggingface.co/v1/production/uploads/1625166923504-noauth.png
Graham Neubig
user
gneubig
64db1421322a5774e09c01e8
Ankush Garg
admin_assigned
2023-08-15T13:21:57.584Z
609bfde2138207cc2f2f3a8c
/avatars/8bb0198645bf8aa4e473bab3b1d76b20.svg
garg
user
ankushgarg
64db1421322a5774e09c01e9
Jonathan H. Clark
admin_assigned
2023-08-15T13:22:40.331Z
6244a009a37b7efa7d335e47
/avatars/465c97aa99e0377b09e2f43813da2701.svg
Jonathan Clark
user
jhclark
64db1421322a5774e09c01ea
Markus Freitag
64db1421322a5774e09c01eb
Orhan Firat
The Devil is in the Errors: Leveraging Large Language Models for
  Fine-grained Machine Translation Evaluation
Automatic evaluation of machine translation (MT) is a critical tool driving
the rapid iterative development of MT systems. While considerable progress has
been made on estimating a single scalar quality score, current metrics lack the
informativeness of more detailed schemes that annotate individual errors, such
as Multidimensional Quality Metrics (MQM). In this paper, we help fill this gap
by proposing AutoMQM, a prompting technique which leverages the reasoning and
in-context learning capabilities of large language models (LLMs) and asks them
to identify and categorize errors in translations. We start by evaluating
recent LLMs, such as PaLM and PaLM-2, through simple score prediction
prompting, and we study the impact of labeled data through in-context learning
and finetuning. We then evaluate AutoMQM with PaLM-2 models, and we find that
it improves performance compared to just prompting for scores (with
particularly large gains for larger models) while providing interpretability
through error spans that align with human annotations.
AutoMQM uses large language models to identify and categorize errors in machine translation, offering interpretability and improving performance over traditional score prediction.
machine translation Multidimensional Quality Metrics MQM large language models LLMs PaLM PaLM-2 score prediction in-context learning finetuning error spans human annotations