CoCoA-MT: A Dataset and Benchmark for Contrastive Controlled MT with Application to Formality
The machine translation (MT) task is typically formulated as that of
returning a single translation for an input segment. However, in many cases,
multiple different translations are valid and the appropriate translation may
depend on the intended target audience, characteristics of the speaker, or even
the relationship between speakers. Specific problems arise when dealing with
honorifics, particularly translating from English into languages with formality
markers. For example, the sentence "Are you sure?" can be translated in German
as "Sind Sie sich sicher?" (formal register) or "Bist du dir sicher?"
(informal). Using wrong or inconsistent tone may be perceived as inappropriate
or jarring for users of certain cultures and demographics. This work addresses
the problem of learning to control target language attributes, in this case
formality, from a small amount of labeled contrastive data. We introduce an
annotated dataset (CoCoA-MT) and an associated evaluation metric for training
and evaluating formality-controlled MT models for six diverse target languages.
We show that we can train formality-controlled models by fine-tuning on labeled
contrastive data, achieving high accuracy (82% in-domain and 73% out-of-domain)
while maintaining overall quality.