You will be shown sections from a scientific research paper, together with a question about the paper. Paragraphs in the paper are separated by newlines. Your task is to answer the question based on the contents of the paper.

Paper:
----------------------------------------
A Multi-Task Architecture on Relevance-based Neural Query Translation

We describe a multi-task learning approach to train a Neural Machine Translation (NMT) model with a Relevance-based Auxiliary Task (RAT) for search query translation. The translation process for Cross-lingual Information Retrieval (CLIR) task is usually treated as a black box and it is performed as an independent step. However, an NMT model trained on sentence-level parallel data is not aware of the vocabulary distribution of the retrieval corpus. We address this problem with our multi-task learning architecture that achieves 16% improvement over a strong NMT baseline on Italian-English query-document dataset. We show using both quantitative and qualitative analysis that our model generates balanced and precise translations with the regularization effect it achieves from multi-task learning paradigm.

Introduction
CLIR systems retrieve documents written in a language that is different from search query language BIBREF0 . The primary objective of CLIR is to translate or project a query into the language of the document repository BIBREF1 , which we refer to as Retrieval Corpus (RC). To this end, common CLIR approaches translate search queries using a Machine Translation (MT) model and then use a monolingual IR system to retrieve from RC. In this process, a translation model is treated as a black box BIBREF2 , and it is usually trained on a sentence level parallel corpus, which we refer to as Translation Corpus (TC).
We address a pitfall of using existing MT models for query translation BIBREF1 . An MT model trained on TC does not have any knowledge of RC. In an extreme setting, where there are no common terms between the target side of TC and RC, a well trained and tested translation model would fail because of vocabulary mismatch between the translated query and documents of RC. Assuming a relaxed scenario where some commonality exists between two corpora, a translation model might still perform poorly, favoring terms that are more likely in TC but rare in RC. Our hypothesis is that a search query translation model would perform better if a translated query term is likely to appear in the both retrieval and translation corpora, a property we call balanced translation.
To achieve balanced translations, it is desired to construct an MT model that is aware of RC vocabulary. Different types of MT approaches have been adopted for CLIR task, such as dictionary-based MT, rule-based MT, statistical MT etc. BIBREF3 . However, to the best of our knowledge, a neural search query translation approach has yet to be taken by the community. NMT models with attention based encoder-decoder techniques have achieved state-of-the-art performance for several language pairs BIBREF4 . We propose a multi-task learning NMT architecture that takes RC vocabulary into account by learning Relevance-based Auxiliary Task (RAT). RAT is inspired from two word embedding learning approaches: Relevance-based Word Embedding (RWE) BIBREF5 and Continuous Bag of Words (CBOW) embedding BIBREF6 . We show that learning NMT with RAT enables it to generate balanced translation.
NMT models learn to encode the meaning of a source sentence and decode the meaning to generate words in a target language BIBREF7 . In the proposed multi-task learning model, RAT shares the decoder embedding and final representation layer with NMT. Our architecture answers the following question: In the decoding stage, can we restrict an NMT model so that it does not only generate terms that are highly likely in TC?. We show that training a strong baseline NMT with RAT roughly achieves 16% improvement over the baseline. Using a qualitative analysis, we further show that RAT works as a regularizer and prohibits NMT to overfit to TC vocabulary.

Balanced Translation Approach
We train NMT with RAT to achieve better query translations. We improve a recently proposed NMT baseline, Transformer, that achieves state-of-the-art results for sentence pairs in some languages BIBREF8 . We discuss Transformer, RAT, and our multi-task learning architecture that achieves balanced translation.

NMT and Transformer
In principle, we could adopt any NMT and combine it with RAT. An NMT system directly models the conditional probability INLINEFORM0 of translating a source sentence, INLINEFORM1 , to a target sentence INLINEFORM2 . A basic form of NMT comprises two components: (a) an encoder that computes the representations or meaning of INLINEFORM3 and (b) a decoder that generates one target word at a time. State-of-the-art NMT models have an attention component that “searches for a set of positions in a source sentence where the most relevant information is concentrated” BIBREF4 .
For this study, we use a state-of-the-art NMT model, Transformer BIBREF8 , that uses positional encoding and self attention mechanism to achieve three benefits over the existing convolutional or recurrent neural network based models: (a) reduced computational complexity of each layer, (b) parallel computation, and (c) path length between long-range dependencies.

Relevance-based Auxiliary Task (RAT)
We define RAT a variant of word embedding task BIBREF6 . Word embedding approaches learn high dimensional dense representations for words and their objective functions aim to capture contextual information around a word. BIBREF5 proposed a model that learns word vectors by predicting words in relevant documents retrieved against a search query. We follow the same idea but use a simpler learning approach that is suitable for our task. They tried to predict words from the relevance model BIBREF9 computed from a query, which does not work for our task because the connection between a query and ranked sentences falls rapidly after the top one (see below).
We consider two data sources for learning NMT and RAT jointly. The first one is a sentence-level parallel corpus, which we refer to as translation corpus, INLINEFORM0 . The second one is the retrieval corpus, which is a collection of INLINEFORM1 documents INLINEFORM2 in the same language as INLINEFORM3 . Our word-embedding approach takes each INLINEFORM4 , uses it as a query to retrieve the top document INLINEFORM5 . After that we obtain INLINEFORM6 by concatenating INLINEFORM7 with INLINEFORM8 and randomly shuffling the words in the combined sequence. We then augment INLINEFORM9 using INLINEFORM10 and obtain a dataset, INLINEFORM11 . We use INLINEFORM12 to learn a continuous bag of words (CBOW) embedding as proposed by BIBREF6 . This learning component shares two layers with the NMT model. The goal is to expose the retrieval corpus' vocabulary to the NMT model. We discuss layer sharing in the next section.
We select the single top document retrieved against a sentence INLINEFORM0 because a sentence is a weak representation of information need. As a result, documents at lower ranks show heavy shift from the context of the sentence query. We verified this by observing that a relevance model constructed from the top INLINEFORM1 documents does not perform well in this setting. We thus deviate from the relevance model based approach taken by BIBREF5 and learn over the random shuffling of INLINEFORM2 and a single document. Random shuffling has shown reasonable effectiveness for word embedding construction for comparable corpus BIBREF10 .

Multi-task NMT Architecture
Our balanced translation architecture is presented in Figure FIGREF3 . This architecture is NMT-model agnostic as we only propose to share two layers common to most NMTs: the trainable target embedding layer and the transformation function BIBREF7 that outputs a probability distribution over the union of the vocabulary of TC and RC. Hence, the size of the vocabulary, INLINEFORM0 , is much larger compared to TC and it enables the model to access RC. In order to show task sharing clearly we placed two shared layers between NMT and RAT in Figure FIGREF3 . We also show the two different paths taken by two different tasks at training time: the NMT path in shown with red arrows while the RAT path is shown in green arrows.
On NMT path training loss is computed as the sum of term-wise softmax with cross-entropy loss of the predicted translation and the human translation and it summed over a batch of sentence pairs, INLINEFORM0 . We also use a similar loss function to train word embedding over a set of context ( INLINEFORM1 ) and pivot ( INLINEFORM2 ) pairs formed using INLINEFORM3 as query to retrieve INLINEFORM4 using Query Likelihood (QL) ranker, INLINEFORM5 . This objective is similar to CBOW word embedding as context is used to predict pivot word Here, we use a scaling factor INLINEFORM6 , to have a balance between the gradients from the NMT loss and RAT loss. For RAT, the context is drawn from a context window following BIBREF6 .
In the figure, INLINEFORM0 and INLINEFORM1 represents the top document retrieved against INLINEFORM2 . The shuffler component shuffles INLINEFORM3 and INLINEFORM4 and creates (context, pivot) pairs. After that those data points are passed through a fully connected linear projection layer and eventually to the transformation function. Intuitively, the word embedding task is similar to NMT as it tries to assign a large probability mass to a target word given a context. However, it enables the transformation function and decoding layer to assign probability mass not only to terms from TC, but also to terms from RC. This implicitly prohibits NMT to overfit and provides a regularization effect. A similar technique was proposed by BIBREF11 to handle out-of-vocabulary or less frequent words for NMT. For these terms they enabled the transformation (also called the softmax cross-entropy layer) to fairly distribute probability mass among similar words. In contrast, we focus on relevant terms rather than similar terms.

Results and Analysis
Table TABREF14 shows the effectiveness of our model (multi-task transformer) over the baseline transformer BIBREF8 . Our model achieves significant performance gains in the test sets over the baseline for both Italian and Finnish query translation. The overall low MAP for NMT can possibly be improved with larger TC. Moreover, our model validation approach requires access to RC index, and it slows down overall training process. Hence, we could not train our model for a large number of epochs - it may be another cause of the low performance.
We want to show that translation terms generated by our multi-task transformer are roughly equally likely to be seen in the Europarl corpus (TC) or the CLEF corpus (RC). Given a translation term INLINEFORM0 , we compute the ratio of the probability of seeing INLINEFORM1 in TC and RC, INLINEFORM2 . Here, INLINEFORM3 and INLINEFORM4 is calculated similarly. Given a query INLINEFORM5 and its translation INLINEFORM6 provided by model INLINEFORM7 , we calculate the balance of INLINEFORM8 , INLINEFORM9 . If INLINEFORM10 is close to 1, the translation terms are as likely in TC as in RC. Figure FIGREF15 shows the balance values for transformer and our model for a random sample of 20 queries from the validation set of Italian queries, respectively. Figure FIGREF16 shows the balance values for transformer and our model for a random sample of 20 queries from the test set of Italian queries, respectively. It is evident that our model achieves better balance compared to baseline transformer, except for a very few cases.
Given a query INLINEFORM0 , consider INLINEFORM1 as the set of terms from human translation of INLINEFORM2 and INLINEFORM3 as the set of translation terms generated by model INLINEFORM4 . We define INLINEFORM5 and INLINEFORM6 as precision and recall of INLINEFORM7 for model INLINEFORM8 . In Table TABREF19 , we report average precision and recall for both transformer and our model across our train and validation query set over two language pairs. Our model generates precise translation, i.e. it avoids terms that might be useless or even harmful for retrieval. Generally, from our observation, avoided terms are highly likely terms from TC and they are generated because of translation model overfitting. Our model achieves a regularization effect through an auxiliary task. This confirms results from existing multi-tasking literature BIBREF15 .
To explore translation quality, consider pair of sample translations provided by two models. For example, against an Italian query, medaglia oro super vinse medaglia oro super olimpiadi invernali lillehammer, translated term set from our model is {gold, coin, super, free, harmonising, won, winter, olympics}, while transformer output is {olympic, gold, one, coin, super, years, won, parliament, also, two, winter}. Term set from human translation is: {super, gold, medal, won, lillehammer, olypmic, winter, games}. Transformer comes up with terms like parliament, also, two and years that never appears in human translation. We found that these terms are very likely in Europarl and rare in CLEF. Our model also generates terms such as harmonising, free, olympics that not generated by transformer. However, we found that these terms are equally likely in Europarl and CLEF.

Conclusion
We present a multi-task learning architecture to learn NMT for search query translation. As the motivating task is CLIR, we evaluated the ranking effectiveness of our proposed architecture. We used sentences from the target side of the parallel corpus as queries to retrieve relevant document and use terms from those documents to train a word embedding model along with NMT. One big challenge in this landscape is to sample meaningful queries from sentences as sentences do not directly convey information need. In the future, we hope to learn models that are able to sample search queries or information needs from sentences and use the output of that model to get relevant documents.

Acknowledgments
This work was supported in part by the Center for Intelligent Information Retrieval and in part by the Air Force Research Laboratory (AFRL) and IARPA under contract #FA8650-17-C-9118 under subcontract #14775 from Raytheon BBN Technologies Corporation. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsor.

Loss Function and Validation Performance Analysis
We show the loss function analysis of transformer and our model. Figure FIGREF23 shows the validation performance of transformer against global training steps. Figure FIGREF21 show the validation performance of our model for the same number of global steps. Figure FIGREF22 shows that NMT loss is going down with the number of steps, while Figure FIGREF20 shows the degradation of the loss of our proposed RAT task.

Figure 1: The architecture of our multi-task NMT. Note that, rectangles indicate data sources and rectangles with rounded corners indicate functions or layers.

Table 1: Statistics of resources used for training. |VF | and |VE | are the vocabulary size for the source language and the target English language, respectively.

Table 2: Results for ranking with query translation models, in terms of MAP.

Figure 2: Balance values of a sample of val queries

Figure 3: Balance values of a sample of test queries

Table 3: Average precision and recall of translated queries, respectively reported in tuples.

Figure 4: RAT loss of our model on Italian-English training data

Figure 5: Validation performance of our model on Italian-English validation data

Figure 6: NMT loss of our model on Italian-English training data

Figure 7: Validation set performance of Transformer on Italian-English training data
----------------------------------------

Question: what quantitative analysis is done?

To answer the question, format your response as a `json` object with two fields:

"answer": A string providing a succinct answer to the question, in your own words.
"evidence": An array of strings. Each entry should be a full paragraph from the paper. Together, the evidence should serve as a justification for the answer.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": "BERT and RoBERTa."
  "evidence": ["We compare our approach against two baselines. In Table 1, we compare against BERT. In Table 2, we compare against RoBERTa. Our findings indicate that our approach improves over both baeslines..."]
}

The "answer" field should be roughly 202 characters in length.

Do not include any text in your response other than the json. If the question is unanswerable given the provided excerpts, respond with the single word "null".

To repeat, the question is: what quantitative analysis is done?

Answer JSON object:
{"answer": "Answer with content missing: (Evaluation section) Given that in CLIR the primary goal is to get a better ranked list of documents against a translated query, we only report Mean Average Precision (MAP).", "evidence": ["Results and Analysis"]}