Do GPTs Produce Less Literal Translations?
Large Language Models (LLMs) such as GPT-3 have emerged as general-purpose
language models capable of addressing many natural language generation or
understanding tasks. On the task of Machine Translation (MT), multiple works
have investigated few-shot prompting mechanisms to elicit better translations
from LLMs. However, there has been relatively little investigation on how such
translations differ qualitatively from the translations generated by standard
Neural Machine Translation (NMT) models. In this work, we investigate these
differences in terms of the literalness of translations produced by the two
systems. Using literalness measures involving word alignment and monotonicity,
we find that translations out of English (E-X) from GPTs tend to be less
literal, while exhibiting similar or better scores on MT quality metrics. We
demonstrate that this finding is borne out in human evaluations as well. We
then show that these differences are especially pronounced when translating
sentences that contain idiomatic expressions.
64741c256a972f252de61c64
Vikas Raunak
admin_assigned
2023-05-31T09:15:45.115Z
5f06175e5d08220171a0ae32
https://cdn-avatars.huggingface.co/v1/production/uploads/1652484966062-5f06175e5d08220171a0ae32.jpeg
Vikas Raunak
user
vyraun
64741c256a972f252de61c65
Arul Menezes
64741c256a972f252de61c66
Matt Post
admin_assigned
2023-05-31T09:17:07.214Z
63fd05b53c880680af41fbf0
/avatars/7c3046c7d72ebe7f0b09726482d3df14.svg
Matt Post
user
mjpost
64741c256a972f252de61c67
Hany Hassan Awadallah
Do GPTs Produce Less Literal Translations?
Large Language Models (LLMs) such as GPT-3 have emerged as general-purpose
language models capable of addressing many natural language generation or
understanding tasks. On the task of Machine Translation (MT), multiple works
have investigated few-shot prompting mechanisms to elicit better translations
from LLMs. However, there has been relatively little investigation on how such
translations differ qualitatively from the translations generated by standard
Neural Machine Translation (NMT) models. In this work, we investigate these
differences in terms of the literalness of translations produced by the two
systems. Using literalness measures involving word alignment and monotonicity,
we find that translations out of English (E-X) from GPTs tend to be less
literal, while exhibiting similar or better scores on MT quality metrics. We
demonstrate that this finding is borne out in human evaluations as well. We
then show that these differences are especially pronounced when translating
sentences that contain idiomatic expressions.
The study compares the literalness of translations generated by large language models like GPT-3 and neural machine translation models, finding that GPT-3 translations are less literal but perform similarly or better on quality metrics.
Large Language Models LLMs GPT-3 Machine Translation MT Neural Machine Translation NMT few-shot prompting word alignment monotonicity idiomatic expressions