You will be shown sections from a scientific research paper, together with a question about the paper. Your task is to answer the question based on the contents of the paper. Your response should be a `json` object with two fields:

"answer": An array of strings extracted directly from the paper which, collectively, answer the question.
"evidence": An array of strings. Each should be an excerpt from the paper, in which one or more of the extracted answers can be found.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": ["BERT", "RoBERTa"],
  "evidence": ["In our experiments, we compare the performance of our model against BERT and RoBERTa."]
}

Do not include any text in your response other than the json.

If the question is unanswerable given the provided excerpts, respond with the single word "null".

Paper: Combining Advanced Methods in Japanese-Vietnamese Neural Machine Translation

Neural machine translation (NMT) systems have recently obtained state-of-the art in many machine translation systems between popular language pairs because of the availability of data. For low-resourced language pairs, there are few researches in this field due to the lack of bilingual data. In this paper, we attempt to build the first NMT systems for a low-resourced language pairs:Japanese-Vietnamese. We have also shown significant improvements when combining advanced methods to reduce the adverse impacts of data sparsity and improve the quality of NMT systems. In addition, we proposed a variant of Byte-Pair Encoding algorithm to perform effective word segmentation for Vietnamese texts and alleviate the rare-word problem that persists in NMT systems.

Introduction
Neural machine translation (NMT) BIBREF0 , BIBREF1 is widely applied for machine translation (MT) in recent years and focuses on popular language pairs such as English INLINEFORM0 French, English INLINEFORM1 German, English INLINEFORM2 Chinese or English INLINEFORM3 Japanese. NMT has obtained state-of-the-art performance on those language pairs compared to the traditional statistical machine translation (SMT) when given enough data BIBREF2 , BIBREF3 . Furthermore, due to the ability of feature learning, NMT systems can be trained end-to-end with pure parallel texts and minimal linguistic knowledge of the languages involved. Thus it makes training NMT for a new language pair much easier, more scalable and robust. Nevertheless, NMT has not been employed in many low-resourced language pairs since in those scenarios, data scarcity often limits the learning ability of neural methods. In contrast, combinating complicated linguistic-driven features in a typical log-linear framework still keeps SMT the best approach in many translation directions but also hard to apply to new domains or to other language pairs.
In this paper we attempt to build NMT systems for such a low-resourced language pair: Japanese INLINEFORM0 Vietnamese. Our aim is to set the first and reasonable NMT systems that can be reproducible in order to serve as the baseline for further researches in the direction. Furthermore, we conduct experiments using some advanced methods to improve the quality of the systems. An important criteria for those methods is that they must be scalable and language-independent as much as possible. The criteria ensures the basic principle of NMT as well as the reproducibility of the systems. On the other hand, the methods are chosen in the direction that they would help alleviate the data sparsity problem of NMT when being applied in this low-resourced setting.
Specifically, to deal with rare-word translation problems, we experiment with translation units in different levels: subword, word and beyond. In morphological-rich languages such as English or German, using subword as the translation units is often suitable since neural methods are able to induce meaning or linguistic function of sub-components constituting a word. Byte-Pair Encoding (BPE) BIBREF4 is a simple unsupervised technique to do subword segmentation and it has great effects when applied to NMT training. Japanese and Vietnamese (and some other Asian languages), however, have different word segmentation issues. Hence, it would be difficult to apply BPE directly to the texts in those languages and build NMT systems for subword translation without any modification. In this paper, we experiment different segmentation methods for both languages and also propose a variant of BPE algorithm to learn translation units for Vietnamese in an unsupervised way.
We also attempt to increase the amount of training data by using back-translated texts or mix-source data just from our small available corpus. Those data augmentation approaches have shown their effectiveness on various NMT systems, especially in under-resourced scenarios. While back translation technique is used to generate synthetic data from monolingual corpora, mix-source technique utilizes human-quality corpora in a multilingual setting, leveraging the transfer learning ability across languages. Both are simple but elegantly model the relevant noise needed in training neural architectures in such low-resourced situations.
The main contributions of this paper are:

Vietnamese Tokenization
Vietnamese From the linguistic point of view, each sequence of characters between two white spaces in Vietnamese texts cannot be considered as a word since it does not always have a full meaning to stand alone. For example, in the sentence “hôm nay là sinh nhật của tôi” (English equivalence: “Today is my birthday”), “hôm” and “nay” are not two words, they together form a word, which means “today”. Nevertheless, “hôm” and “nay” somehow still bear some meaning: “hôm”-“day”, “nay”-“now”. Similarly, “sinh”-“birth” and “nhật”-“date” also form the word “sinh nhật”-“birthday” but they are not two distinct words. We could also call them subwords.
In many Vietnamese processing tasks such as Part-of-Speech Tagging, Syntax Parsing or Chunking, there requires a step to concatenate those subwords to make a word since in those tasks, word is necessarily considered as the smallest unit to be processed. This step is normally referred to as word segmentation. There are various word segmentation methods; The best ones are using machine learning approaches to learn from a labeled corpus. It makes the tasks hard and expensive to be applied in other domains. Furthermore, the translation unit in Machine Translation does not need to be a word but can be a subword or sequence of subwords if it have its own meaning.
With this observation in mind, if we consider a subword, i.e. the sequence of characters between two white spaces, as a byte and a sentence as a sequence of bytes, we can apply the BPE algorithm straight-forward: We iteratively find and concatenate the most frequent pair of subwords ( INLINEFORM0 , INLINEFORM1 ) and replace it by an unseen subwords INLINEFORM2 . We do not merge INLINEFORM3 with INLINEFORM4 if one of them are digits or punctuations or other special symbols. The BPE learning algorithm has an arguments which is the minimum value of frequency. In practice, we set the minimum frequency is 2. Listing 1 presents this variant of BPE, which we call VNBPE.
mystyle def get_most_freq_pair(text,min_freq): dict = {} dumpWord = {set_of_separate_symbols} for line in text: w1 = the_first_word_in_the_line for w2 = each_word_in_the_line: if w1,w2 not in dumpWord and w1 is not a number: dict[w1,w2] += 1 get_next_pair_in_the_line() w1 = w2 return (all pairs has freq > min_freq) def update_pairs(pair,text,codes_file): original_word = pair[0] + " " + pair[1] replaced_word = pair[0] + "_" + pair[1] input_file.replace(rew,orw) write_replaced_word_to_codes_file() return input_file ### MAIN PART ### min_freq=2 open(input_file) to read open(codes_file) and (output_file) to write pairs = get_most_freq_pair(input_file,min_freq) arrange_pairs_for_decreasing_of_frequency() for each (freq,pair) in pairs: update_pairs(pair,input_file,codes_file) output_file=input_file close_all_files() Compared to other word segmentation methods which require training on labeled data, our VNBPE is a simple unsupervised method, alike to its original BPE algorithm. Table TABREF7 shows the outputs of one decent word segmentation method and our VNBPE.

Back Translation
One of the approaches to leverage the monolingual data is to use a machine translation system to translate those data in order to create a synthetic parallel data. Normally, the monolingual data in the target language is translated, thus the name of the method: Back Translation BIBREF11 .
More specific, to generate the data for an X INLINEFORM0 Y NMT system, we use the best Y INLINEFORM1 X translation system we have to translate every sentence INLINEFORM2 in the monolingual data of language Y into sentences INLINEFORM3 in the source language X. Then we pair INLINEFORM4 to get the synthetic data. Finally, original bilingual data and synthetic data are mixed to train our NMT from the start.
Back Translation can improve the estimate conditional probability of the target word on the previous context words through adding a bilingual data with approximate translations to the source languages. Furthermore, the synthetic data might contain some translation noise from the Back Translation system, and if this noise is relevant, our NMT can be more robust in learning how to translate noisy inputs. One the other hand, if the quality of the Back Translation system is not adequate, using the synthesis data might bring adverse effects to our NMT.
In this paper, we subsample an amount of Vietnamese monolingual data so that we can create a synthetic corpus having the same size with the Japanese-Vietnamese parallel corpus. In the end, the data we have is double in size compared to the original one.

Mix-Source Approach
Another data augmentation method considered useful in this low-resourced setting is the mix-source method BIBREF12 . In this method, we can utilize the monolingual data of the target language in a multilingual NMT system by mixing the original source sentences with those target monolingual data. The multilingual framework then uses the share information across source and target languages to improve the decision of the target words to be chosen.
Specifically, there are a small parallel corpus INLINEFORM0 of the language pair X-Y which has INLINEFORM1 sentence pairs INLINEFORM2 ( INLINEFORM3 ) and a big monolingual corpus INLINEFORM4 of the language Y which has INLINEFORM5 sentences INLINEFORM6 ( INLINEFORM7 , INLINEFORM8 ). Now from the monolingual corpus INLINEFORM9 we can generate a parallel corpus INLINEFORM10 where we try to model the identical translation Y-Y: INLINEFORM11 ( INLINEFORM12 ). Then we mix INLINEFORM13 and INLINEFORM14 to get a parallel corpus of the size INLINEFORM15 . Similar to the Back Translation, we subsample INLINEFORM16 Vietnamese sentence pairs from the corpus INLINEFORM17 then the size of the parallel data we have is also doubled.
To let the NMT knows which language a certain source sentence is in and then can model the language information, we follows the conventions from BIBREF12 . We append language tags to every word in both source and target sentences of the mixed INLINEFORM0 corpus to indicate the language of the words. This technique shows the effectiveness in low-resourced scenarios BIBREF13 , BIBREF14 and our Japanese INLINEFORM1 Vietnamese is such a scenario.

Preprocessing
After using KyTea to tokenize the Japanese texts, we learn and apply Sennrich's BPE from the tokenized texts. For Vietnamese texts, first we use Moses scripts to normalize the texts from digits, punctuations and special symbols. We use pyvi for Vietnamese word segmentation since it is one of the best tools for this task in term of speed, robustness and performance. On the other hand, we useVNBPE as an alternative way of doing word segmentation. Those two approaches are compared in an extrinsic evaluation of the NMT systems employing them.

Results
We evaluate the quality of translation of systems based on different approaches mentioned in previous sections. The multi-BLEU from Moses scripts is used. The results have shown in the Table TABREF21 .
Baseline. For the baseline systems, the training data includes KyTea-segmented Japanese texts and pyvi-segmented Vietnamese texts. For comparison purpose, we build two baseline systems for each direction: one is use the traditional phrase-based statistical machine translation (SMT), the other one is the NMT system. Although our training set is small but we find that the NMT systems (2) are still more effective than the phrase-based SMT models (1) in both translation directions.
Subword NMT. We applied VNBPE and JPBPE to the baseline's data and trained NMT systems. On Vietnamese INLINEFORM0 Japanese, we observed an improvement of 0.6 BLEU points when we used our VNBPE (3) instead of the pyvi's word segmentation (2). Furthermore, when we trained our NMT models using both BPE methods (4), we obtained a bigger gain of 1.15 BLEU points. The similar improvements can be found in the Japanese INLINEFORM1 Vietnamese as well: 0.29 BLEU points between (3) and (2) and 0.57 BLEU points between (4) and (3). This draws two conclusions: (i), despite using an unsupervised Vietnamese word segmentation which is fast, robust and does not require linguistic resources, our NMT systems performed better than those systems employing a complicate word segmentation method, (ii) BPE works significantly well for Japanese texts after we tokenized the texts.
Data Augmentation. We use the best system for Vietnamese INLINEFORM0 Japanese, which is the NMT systems trained on BPE-processed texts, to generate the synthetic data for Japanese INLINEFORM1 Vietnamese translation. Although we achieved some gain (from 9.04 to 9.39 BLEU points), the effectiveness of Back Translation is not on par with its application on the translation systems of other language pairs. Looking into the Vietnamese INLINEFORM2 Japanese translation of DongDu corpus and its BLEU score, we speculate that it is because the Vietnamese INLINEFORM3 Japanese system is not good enough to produce reasonable synthetic data. In the meanwhile, combining Back Translation and Mix-Source brings a considerable improvements of 0.6 BLEU points compared to not using them.

Related Works
Japanese-Vietnamese MT is firstly mentioned in 2005 BIBREF18 . The authors focused on the difference from embedding structures between Japanese and Vietnamese, and then proposed rules for MT system and experiment on very small dataset (714 Japanese embedding sentences). This approach is suitable for small system applied in a specific domain or language, but it is not easily extendable to other domains or languages due to the expensiveness of building such rules.
The other previous work for Japanese INLINEFORM0 Vietnamese uses SMT BIBREF19 . They also conducted the experiments on parallel corpora collected from TED talks. They used phrase-based and tree-to-string models and have shown that the SMT system trained on French INLINEFORM1 Vietnamese obtains better results than the system of Japanese INLINEFORM2 Vietnamese because French and Vietnamese have more similarities in the structures of sentences than between Japanese and Vietnamese. We also built phrase-based systems on the TED data and achieved better BLEU scores when using NMT.
Recently, some works use monolingual data to improve the accuracy of NMT systems. BIBREF11 have shown significant improvements by using monolingual data on target-side to generate synthetic data and then add them to original training data. BIBREF20 have shown significant improvements by "self-learning" method to generate the target sentences based on monolingual data on the source-side and then combined them with original bilingual data to train. BIBREF12 convert monolingual corpus on the target-side into a bitexts by copying target sentences to the source sentence and then combined original bilingual data together on training. Our systems employ those approaches to exploit monolingual data and show the improved performance for Japanese INLINEFORM0 Vietnamese translations.

Question: what was the baseline?
{"answer": ["traditional phrase-based statistical machine translation (SMT)", "NMT system"], "evidence": ["Baseline. For the baseline systems, the training data includes KyTea-segmented Japanese texts and pyvi-segmented Vietnamese texts. For comparison purpose, we build two baseline systems for each direction: one is use the traditional phrase-based statistical machine translation (SMT), the other one is the NMT system. Although our training set is small but we find that the NMT systems (2) are still more effective than the phrase-based SMT models (1) in both translation directions."]}