You will be shown sections from a scientific research paper, together with a question about the paper. Your task is to answer the question based on the contents of the paper. Your response should be a `json` object with two fields:

"answer": An array of strings extracted directly from the paper which, collectively, answer the question.
"evidence": An array of strings. Each should be an excerpt from the paper, in which one or more of the extracted answers can be found.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": ["BERT", "RoBERTa"],
  "evidence": ["In our experiments, we compare the performance of our model against BERT and RoBERTa."]
}

Do not include any text in your response other than the json.

If the question is unanswerable given the provided excerpts, respond with the single word "null".

Paper: Combining Advanced Methods in Japanese-Vietnamese Neural Machine Translation

Neural machine translation (NMT) systems have recently obtained state-of-the art in many machine translation systems between popular language pairs because of the availability of data. For low-resourced language pairs, there are few researches in this field due to the lack of bilingual data. In this paper, we attempt to build the first NMT systems for a low-resourced language pairs:Japanese-Vietnamese. We have also shown significant improvements when combining advanced methods to reduce the adverse impacts of data sparsity and improve the quality of NMT systems. In addition, we proposed a variant of Byte-Pair Encoding algorithm to perform effective word segmentation for Vietnamese texts and alleviate the rare-word problem that persists in NMT systems.

Introduction
Neural machine translation (NMT) BIBREF0 , BIBREF1 is widely applied for machine translation (MT) in recent years and focuses on popular language pairs such as English INLINEFORM0 French, English INLINEFORM1 German, English INLINEFORM2 Chinese or English INLINEFORM3 Japanese. NMT has obtained state-of-the-art performance on those language pairs compared to the traditional statistical machine translation (SMT) when given enough data BIBREF2 , BIBREF3 . Furthermore, due to the ability of feature learning, NMT systems can be trained end-to-end with pure parallel texts and minimal linguistic knowledge of the languages involved. Thus it makes training NMT for a new language pair much easier, more scalable and robust. Nevertheless, NMT has not been employed in many low-resourced language pairs since in those scenarios, data scarcity often limits the learning ability of neural methods. In contrast, combinating complicated linguistic-driven features in a typical log-linear framework still keeps SMT the best approach in many translation directions but also hard to apply to new domains or to other language pairs.
In this paper we attempt to build NMT systems for such a low-resourced language pair: Japanese INLINEFORM0 Vietnamese. Our aim is to set the first and reasonable NMT systems that can be reproducible in order to serve as the baseline for further researches in the direction. Furthermore, we conduct experiments using some advanced methods to improve the quality of the systems. An important criteria for those methods is that they must be scalable and language-independent as much as possible. The criteria ensures the basic principle of NMT as well as the reproducibility of the systems. On the other hand, the methods are chosen in the direction that they would help alleviate the data sparsity problem of NMT when being applied in this low-resourced setting.
Specifically, to deal with rare-word translation problems, we experiment with translation units in different levels: subword, word and beyond. In morphological-rich languages such as English or German, using subword as the translation units is often suitable since neural methods are able to induce meaning or linguistic function of sub-components constituting a word. Byte-Pair Encoding (BPE) BIBREF4 is a simple unsupervised technique to do subword segmentation and it has great effects when applied to NMT training. Japanese and Vietnamese (and some other Asian languages), however, have different word segmentation issues. Hence, it would be difficult to apply BPE directly to the texts in those languages and build NMT systems for subword translation without any modification. In this paper, we experiment different segmentation methods for both languages and also propose a variant of BPE algorithm to learn translation units for Vietnamese in an unsupervised way.
We also attempt to increase the amount of training data by using back-translated texts or mix-source data just from our small available corpus. Those data augmentation approaches have shown their effectiveness on various NMT systems, especially in under-resourced scenarios. While back translation technique is used to generate synthetic data from monolingual corpora, mix-source technique utilizes human-quality corpora in a multilingual setting, leveraging the transfer learning ability across languages. Both are simple but elegantly model the relevant noise needed in training neural architectures in such low-resourced situations.
The main contributions of this paper are:

Neural Machine Translation
In this section, we will describe the general architecture of NMT as a kind of sequence-to-sequence modeling framework. In this kind of sequence-to-sequence modeling framework, often there is an encoder trying to encode context information from the input sequence and a decoder to generate one item of the output sequence at a time based on the context of both input and output sequences. Besides, an additional component, named attention, exists in between, deciding which parts of the input sequence the decoder should pay attention in order to choose which to output next. In other words, this attention component calculates the context relevant to the decision of the decoder at the considering time. Those components as a whole constitute a large trainable neural architecture called the famous attention-based encoder-decoder framework. This framework becomes popular in many sequence-to-sequence tasks.
In the field of machine translation, using the attention-based encoder-decoder framework is referred to as Neural Machine Translation approach. First presented in BIBREF0 , the encoder and decoder in NMT are recurrent-based, which each hidden unit in those components is a recurrent unit like Long Short-term Memory (LSTM) BIBREF5 or Gated Recurrent Unit(GRU) BIBREF6 . Later, the encoder or decoder can also be a convolutional architecture, as in BIBREF7 . Recently, BIBREF8 introduces transformer architecture, in which both the encoder and decoder are a special variant of attention mechanism, called self attention. In this paper, we will briefly explain the Recurrent NMT as we utilize it in our experiments.
The Recurrent NMT model follows the attention-based architecture proposed by BIBREF0 . The bidirectional recurrent encoder reads every words INLINEFORM0 of a source sentence INLINEFORM1 and encodes a representation INLINEFORM2 of the sentence into a fixed-length vector INLINEFORM3 concatenated from those of the forward and backward directions: INLINEFORM4 
Here INLINEFORM0 is the one-hot vector of the word INLINEFORM1 and INLINEFORM2 is the word embedding matrix which is shared across the source words. INLINEFORM3 is the recurrent unit computing the current hidden state of the encoder based on the previous hidden state. INLINEFORM4 is then called an annotation vector, which represent the information of the source sentence up to the time INLINEFORM5 from both forward and backward directions.
Those annotation vectors of the source sentences are combined in the attention layers in a way that the resulted vector encodes the source context relevant to the considering target word the decoder should produce. Intuitively, a relevance between the previous target word INLINEFORM0 and the annotation vectors INLINEFORM1 corresponding to the source words INLINEFORM2 can be used to form some attention scenario: DISPLAYFORM0 
This specific attention mechanism, originally called alignment model in BIBREF0 , has been employed as a simple feedforward network with the first layer is a learnable layer via adaptation factors INLINEFORM0 , INLINEFORM1 and INLINEFORM2 . The relevance scores INLINEFORM3 are then normalized into attention weights INLINEFORM4 and the context vector INLINEFORM5 is calculated as the weighted sum of all annotation vectors INLINEFORM6 . Depending on how much attention the target word at time INLINEFORM7 put on the source states INLINEFORM8 , a soft alignment is learned and a source context INLINEFORM9 at time INLINEFORM10 is calculated prior to the prediction of the decoder.
Similar to the encoder, the recurrent decoder recursively generates one target word INLINEFORM0 to form a translated target sentence INLINEFORM1 in the end. At time INLINEFORM2 , it takes the previous hidden state of the decoder INLINEFORM3 , the previous embedded word representation INLINEFORM4 and the time-specific context vector INLINEFORM5 as inputs to calculate the current hidden state INLINEFORM6 : INLINEFORM7 
Again, INLINEFORM0 is the recurrent activation function of the decoder and INLINEFORM1 is the shared word embedding matrix of the target sentences.
Given the parallel corpus consisting of INLINEFORM0 training examples INLINEFORM1 , the objective is to maximize the conditional log-probability of the correct translation given a source sentence with respect to the parameters INLINEFORM2 of the whole model: INLINEFORM3 

Japanese Tokenization
In a Japanese written text, there could be a mixture of three different types of scripts: Chinese characters (kanji) and the other two syllabic scripts: hiragana and katakana. Each of kanji characters can be loosely considered as a subword that we mentioned in the previous section. In the meanwhile, each of hiragana or katakana characters can be considered as a latin character in English or Vietnamese. In addition, there is no space in the Japanese written texts to separate the characters, either kanji, ,hiragana or katakana. So we cannot learn good subwords from a small corpus by directly applying BPE or the variant VNBPE on Japanese written texts. In order to learn good subwords and with a little knowledge about Japanese, we decided to use KyTea BIBREF10 to do Japanese word segmentation and then apply Sennrich's BPE on those word-segmented texts. Some examples of the Japanese words going through the word segmentation and BPE are shown in Table TABREF9 .

Data Augmentation
In this section, we describe the data augmentation methods we use to increase the amount of training data in order to make our NMT systems suffer less from the low-resourced situation in Japanese INLINEFORM0 Vietnamese translation. Although NMT systems can predict and generate the translation of unseen words on their vocabularies, but they only perform this well if the parallel corpus for training are sufficiently large. For many under-resourced languages, unfortunately, it hardly presents. In reality, although the monolingual data of Vietnamese and Japanese are immensely available due to the popularity of their speakers, the bilingual Japanese-Vietnamese corpora are very limited and often in low quality or in narrowly technical domains. Therefore, data augmentation methods to exploit monolingual data for NMT systems are necessary to obtain more bilingual data, thus upgrading the translating quality.

Back Translation
One of the approaches to leverage the monolingual data is to use a machine translation system to translate those data in order to create a synthetic parallel data. Normally, the monolingual data in the target language is translated, thus the name of the method: Back Translation BIBREF11 .
More specific, to generate the data for an X INLINEFORM0 Y NMT system, we use the best Y INLINEFORM1 X translation system we have to translate every sentence INLINEFORM2 in the monolingual data of language Y into sentences INLINEFORM3 in the source language X. Then we pair INLINEFORM4 to get the synthetic data. Finally, original bilingual data and synthetic data are mixed to train our NMT from the start.
Back Translation can improve the estimate conditional probability of the target word on the previous context words through adding a bilingual data with approximate translations to the source languages. Furthermore, the synthetic data might contain some translation noise from the Back Translation system, and if this noise is relevant, our NMT can be more robust in learning how to translate noisy inputs. One the other hand, if the quality of the Back Translation system is not adequate, using the synthesis data might bring adverse effects to our NMT.
In this paper, we subsample an amount of Vietnamese monolingual data so that we can create a synthetic corpus having the same size with the Japanese-Vietnamese parallel corpus. In the end, the data we have is double in size compared to the original one.

Data Preparation
We collected Japanese-Vietnamese parallel data from TED talks extracted from WIT3's corpus BIBREF15 . After removing blank and duplicate lines we obtained 106758 pairs of sentences. The validation set used in all experiments is dev2010 and the test set is tst2010.
The data augmentation methods has been applied only for the Japanese INLINEFORM0 Vietnamese direction. For Back Translation, we use Vietnamese monolingual data from VNESEcorpus of DongDu which includes 349578 sentences. We shuffle the lines of VNESEcorpus corpus and take out the first 106758 sentences (the same as the number of sentence pairs in the original parallel corpus). For Mix-Source, instead of using a subsampled monolingual corpus, we use the Vietnamese part of the Japanese-Vietnamese parallel corpus in order to learn the multilingual information in the same domain. Our datasets are listed in Table TABREF14 .

System Architecture and Training
We implement the translation systems using OpenNMT-py framework BIBREF16 . Our system architecture includes two bi-directional LSTM layers for the encoder and two LSTM layers for the decoder, each layer has the size of 512 hidden units. The size of source and target embedding layers is also 512. We use Adam optimizer BIBREF17 and learning rate annealing scheme with the initial learning rate at INLINEFORM0 . We train each systems for 15 epochs with the batch size of 64. The best model in term of the unigram accuracy on the validation set is usually used to translate the test set with beam size of 16. Other settings are the default settings of OpenNMT-py, otherwise already noted.

Conclusion
We has built the first Japanese INLINEFORM0 Vietnamese NMT systems and released the dataset as well as the associated training scripts. We have also shown that the proposed VNBPE algorithm can be used for Vietnamese word segmentation in order to conduct neural machine translation. Furthermore, by adapting Back Translation and Mix-Source, our NMT systems achieved the best improvement on the dataset. In the future, we will exploit more domain and multilingual information to improve the quality of the systems.

Table III: Evaluation of Japanese↔Vietnamese NMT systems.

Question: what japanese-vietnamese dataset do they use?
{"answer": ["WIT3\u0027s corpus"], "evidence": ["We collected Japanese-Vietnamese parallel data from TED talks extracted from WIT3\u0027s corpus BIBREF15 . After removing blank and duplicate lines we obtained 106758 pairs of sentences. The validation set used in all experiments is dev2010 and the test set is tst2010."]}