On Evaluating Multilingual Compositional Generalization with Translated Datasets
Compositional generalization allows efficient learning and human-like
inductive biases. Since most research investigating compositional
generalization in NLP is done on English, important questions remain
underexplored. Do the necessary compositional generalization abilities differ
across languages? Can models compositionally generalize cross-lingually? As a
first step to answering these questions, recent work used neural machine
translation to translate datasets for evaluating compositional generalization
in semantic parsing. However, we show that this entails critical semantic
distortion. To address this limitation, we craft a faithful rule-based
translation of the MCWQ dataset from English to Chinese and Japanese. Even with
the resulting robust benchmark, which we call MCWQ-R, we show that the
distribution of compositions still suffers due to linguistic divergences, and
that multilingual models still struggle with cross-lingual compositional
generalization. Our dataset and methodology will be useful resources for the
study of cross-lingual compositional generalization in other tasks.