On Non-interactive Evaluation of Animal Communication Translators
If you had an AI Whale-to-English translator, how could you validate whether
or not it is working? Does one need to interact with the animals or rely on
grounded observations such as temperature? We provide theoretical and
proof-of-concept experimental evidence suggesting that interaction and even
observations may not be necessary for sufficiently complex languages. One may
be able to evaluate translators solely by their English outputs, offering
potential advantages in terms of safety, ethics, and cost. This is an instance
of machine translation quality evaluation (MTQE) without any reference
translations available. A key challenge is identifying ``hallucinations,''
false translations which may appear fluent and plausible. We propose using
segment-by-segment translation together with the classic NLP shuffle test to
evaluate translators. The idea is to translate animal communication, turn by
turn, and evaluate how often the resulting translations make more sense in
order than permuted. Proof-of-concept experiments on data-scarce human
languages and constructed languages demonstrate the potential utility of this
evaluation methodology. These human-language experiments serve solely to
validate our reference-free metric under data scarcity. It is found to
correlate highly with a standard evaluation based on reference translations,
which are available in our experiments. We also perform a theoretical analysis
suggesting that interaction may not be necessary nor efficient in the early
stages of learning to translate.
68f7428b24c4489363111c12
Orr Paradise
68f7428b24c4489363111c13
David F. Gruber
68f7428b24c4489363111c14
Adam Tauman Kalai
On Non-interactive Evaluation of Animal Communication Translators
If you had an AI Whale-to-English translator, how could you validate whether
or not it is working? Does one need to interact with the animals or rely on
grounded observations such as temperature? We provide theoretical and
proof-of-concept experimental evidence suggesting that interaction and even
observations may not be necessary for sufficiently complex languages. One may
be able to evaluate translators solely by their English outputs, offering
potential advantages in terms of safety, ethics, and cost. This is an instance
of machine translation quality evaluation (MTQE) without any reference
translations available. A key challenge is identifying ``hallucinations,''
false translations which may appear fluent and plausible. We propose using
segment-by-segment translation together with the classic NLP shuffle test to
evaluate translators. The idea is to translate animal communication, turn by
turn, and evaluate how often the resulting translations make more sense in
order than permuted. Proof-of-concept experiments on data-scarce human
languages and constructed languages demonstrate the potential utility of this
evaluation methodology. These human-language experiments serve solely to
validate our reference-free metric under data scarcity. It is found to
correlate highly with a standard evaluation based on reference translations,
which are available in our experiments. We also perform a theoretical analysis
suggesting that interaction may not be necessary nor efficient in the early
stages of learning to translate.
Theoretical and experimental evidence suggests that evaluating AI translators for complex languages can be done solely through their outputs, using a segment-by-segment translation and shuffle test to identify hallucinations and assess quality without reference translations.
machine translation quality evaluation MTQE hallucinations segment-by-segment translation shuffle test data-scarce human languages constructed languages theoretical analysis