BigVideo: A Large-scale Video Subtitle Translation Dataset for Multimodal Machine Translation
We present a large-scale video subtitle translation dataset, BigVideo, to
facilitate the study of multi-modality machine translation. Compared with the
widely used How2 and VaTeX datasets, BigVideo is more than 10 times larger,
consisting of 4.5 million sentence pairs and 9,981 hours of videos. We also
introduce two deliberately designed test sets to verify the necessity of visual
information: Ambiguous with the presence of ambiguous words, and Unambiguous in
which the text context is self-contained for translation. To better model the
common semantics shared across texts and videos, we introduce a contrastive
learning method in the cross-modal encoder. Extensive experiments on the
BigVideo show that: a) Visual information consistently improves the NMT model
in terms of BLEU, BLEURT, and COMET on both Ambiguous and Unambiguous test
sets. b) Visual information helps disambiguation, compared to the strong text
baseline on terminology-targeted scores and human evaluation. Dataset and our
implementations are available at https://github.com/DeepLearnXMU/BigVideo-VMT.