You will be shown sections from a scientific research paper, together with a question about the paper. Your task is to answer the question based on the contents of the paper. Your response should be a `json` object with two fields:

"answer": An array of strings extracted directly from the paper which, collectively, answer the question.
"evidence": An array of strings. Each should be an excerpt from the paper, in which one or more of the extracted answers can be found.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": ["BERT", "RoBERTa"],
  "evidence": ["In our experiments, we compare the performance of our model against BERT and RoBERTa."]
}

Do not include any text in your response other than the json.

If the question is unanswerable given the provided excerpts, respond with the single word "null".

Paper: Generating Descriptions from Structured Data Using a Bifocal Attention Mechanism and Gated Orthogonalization

In this work, we focus on the task of generating natural language descriptions from a structured table of facts containing fields (such as nationality, occupation, etc) and values (such as Indian, actor, director, etc). One simple choice is to treat the table as a sequence of fields and values and then use a standard seq2seq model for this task. However, such a model is too generic and does not exploit task-specific characteristics. For example, while generating descriptions from a table, a human would attend to information at two levels: (i) the fields (macro level) and (ii) the values within the field (micro level). Further, a human would continue attending to a field for a few timesteps till all the information from that field has been rendered and then never return back to this field (because there is nothing left to say about it). To capture this behavior we use (i) a fused bifocal attention mechanism which exploits and combines this micro and macro level information and (ii) a gated orthogonalization mechanism which tries to ensure that a field is remembered for a few time steps and then forgotten. We experiment with a recently released dataset which contains fact tables about people and their corresponding one line biographical descriptions in English. In addition, we also introduce two similar datasets for French and German. Our experiments show that the proposed model gives 21% relative improvement over a recently proposed state of the art method and 10% relative improvement over basic seq2seq models. The code and the datasets developed as a part of this work are publicly available.

Related work
Natural Language Generation has always been of interest to the research community and has received a lot of attention in the past. The approaches for NLG range from (i) rule based approaches (e.g., BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 ) (ii) modular statistical approaches which divide the process into three phases (planning, selection and surface realization) and use data driven approaches for one or more of these phases BIBREF15 , BIBREF16 , BIBREF17 , BIBREF18 , BIBREF19 (iii) hybrid approaches which rely on a combination of handcrafted rules and corpus statistics BIBREF20 , BIBREF21 , BIBREF22 and (iv) the more recent neural network based models BIBREF1 .
Neural models for NLG have been proposed in the context of various tasks such as machine translation BIBREF1 , document summarization BIBREF2 , BIBREF4 , paraphrase generation BIBREF23 , image captioning BIBREF24 , video summarization BIBREF25 , query based document summarization BIBREF5 and so on. Most of these models are data hungry and are trained on large amounts of data. On the other hand, NLG from structured data has largely been studied in the context of small datasets such as WeatherGov BIBREF26 , RoboCup BIBREF27 , NFL Recaps BIBREF15 , Prodigy-Meteo BIBREF28 and TUNA Challenge BIBREF29 . Recently weather16 proposed RNN/LSTM based neural encoder-decoder models with attention for WeatherGov and RoboCup datasets.
Unlike the datasets mentioned above, the biography dataset introduced by lebret2016neural is larger (700K {table, descriptions} pairs) and has a much larger vocabulary (400K words as opposed to around 350 or fewer words in the above datasets). Further, unlike the feed-forward neural network based model proposed by BIBREF0 we use a sequence to sequence model and introduce components to address the peculiar characteristics of the task. Specifically, we introduce neural components to address the need for attention at two levels and to address the stay on and never look back behaviour required by the decoder. KiddonZC16 have explored the use of checklists to track previously visited ingredients while generating recipes from ingredients. Note that two-level attention mechanisms have also been used in the context of summarization BIBREF6 , document classification BIBREF7 , dialog systems BIBREF8 , etc. However, these works deal with unstructured data (sentences at the higher level and words at a lower level) as opposed to structured data in our case.

Fused Bifocal Attention Mechanism
Intuitively, when a human writes a description from a table she keeps track of information at two levels. At the macro level, it is important to decide which is the appropriate field to attend to next and at a micro level (i.e., within a field) it is important to know which values to attend to next. To capture this behavior, we use a bifocal attention mechanism as described below.
Macro Attention: Consider the INLINEFORM0 -th field INLINEFORM1 which has values INLINEFORM2 . Let INLINEFORM3 be the representation of this field in the infobox. This representation can either be (i) the word embedding of the field name or (ii) some function INLINEFORM4 of the values in the field or (iii) a concatenation of (i) and (ii). The function INLINEFORM5 could simply be the sum or average of the embeddings of the values in the field. Alternately, this function could be a GRU (or LSTM) which treats these values within a field as a sequence and computes the field representation as the final representation of this sequence (i.e., the representation of the last time-step). We found that bidirectional GRU is a better choice for INLINEFORM6 and concatenating the embedding of the field name with this GRU representation works best. Further, using a bidirectional GRU cell to take contextual information from neighboring fields also helps (these are the orange colored cells in the top-left block in Figure FIGREF3 with macro attention). Given these representations INLINEFORM7 for all the INLINEFORM8 fields we compute an attention over the fields (macro level). DISPLAYFORM0 
where INLINEFORM0 is the state of the decoder at time step INLINEFORM1 . INLINEFORM2 and INLINEFORM3 are parameters, INLINEFORM4 is the total number of fields in the input, INLINEFORM5 is the macro (field level) context vector at the INLINEFORM6 -th time step of the decoder.
Micro Attention: Let INLINEFORM0 be the representation of the INLINEFORM1 -th value in a given field. This representation could again either be (i) simply the embedding of this value (ii) or a contextual representation computed using a function INLINEFORM2 which also considers the other values in the field. For example, if INLINEFORM3 are the values in a field then these values can be treated as a sequence and the representation of the INLINEFORM4 -th value can be computed using a bidirectional GRU over this sequence. Once again, we found that using a bi-GRU works better then simply using the embedding of the value. Once we have such a representation computed for all values across all the fields, we compute the attention over these values (micro level) as shown below : DISPLAYFORM0 
 where INLINEFORM0 is the state of the decoder at time step INLINEFORM1 . INLINEFORM2 and INLINEFORM3 are parameters, INLINEFORM4 is the total number of values across all the fields.
Fused Attention: Intuitively, the attention weights assigned to a field should have an influence on all the values belonging to the particular field. To ensure this, we reweigh the micro level attention weights based on the corresponding macro level attention weights. In other words, we fuse the attention weights at the two levels as: DISPLAYFORM0 
 where INLINEFORM0 is the field corresponding to the INLINEFORM1 -th value, INLINEFORM2 is the macro level context vector.

Experimental setup
We now describe our experimental setup:

Datasets
We use the WikiBio dataset introduced by lebret2016neural. It consists of INLINEFORM0 biography articles from English Wikipedia. A biography article corresponds to a person (sportsman, politician, historical figure, actor, etc.). Each Wikipedia article has an accompanying infobox which serves as the structured input and the task is to generate the first sentence of the article (which typically is a one-line description of the person). We used the same train, valid and test sets which were made publicly available by lebret2016neural.
We also introduce two new biography datasets, one in French and one in German. These datasets were created and pre-processed using the same procedure as outlined in lebret2016neural. Specifically, we extracted the infoboxes and the first sentence from the corresponding Wikipedia article. As with the English dataset, we split the French and German datasets randomly into train (80%), test (10%) and valid (10%). The French and German datasets extracted by us has been made publicly available. The number of examples was 170K and 50K and the vocabulary size was 297K and 143K for French and German respectively. Although in this work we focus only on generating descriptions in one language, we hope that this dataset will also be useful for developing models which jointly learn to generate descriptions from structured data in multiple languages.

Models compared
We compare with the following models:
1. BIBREF0 : This is a conditional language model which uses a feed-forward neural network to predict the next word in the description conditioned on local characteristics (i.e., words within a field) and global characteristics (i.e., overall structure of the infobox).
2. BIBREF9 : This model was proposed in the context of the WeatherGov and RoboCup datasets which have a much smaller vocabulary. They use an improved attention model with additional regularizer terms which influence the weights assigned to the fields.
3. Basic Seq2Seq: This is the vanilla encode-attend-decode model BIBREF1 . Further, to deal with the large vocabulary ( INLINEFORM0 400K words) we use a copying mechanism as a post-processing step. Specifically, we identify the time steps at which the decoder produces unknown words (denoted by the special symbol UNK). For each such time step, we look at the attention weights on the input words and replace the UNK word by that input word which has received maximum attention at this timestep. This process is similar to the one described in BIBREF30 . Even lebret2016neural have a copying mechanism tightly integrated with their model.

Hyperparameter tuning
We tuned the hyperparameters of all the models using a validation set. As mentioned earlier, we used a bidirectional GRU cell as the function INLINEFORM0 for computing the representation of the fields and the values (see Section SECREF4 ). For all the models, we experimented with GRU state sizes of 128, 256 and 512. The total number of unique words in the corpus is around 400K (this includes the words in the infobox and the descriptions). Of these, we retained only the top 20K words in our vocabulary (same as BIBREF0 ). We initialized the embeddings of these words with 300 dimensional Glove embeddings BIBREF31 . We used Adam BIBREF32 with a learning rate of INLINEFORM1 , INLINEFORM2 and INLINEFORM3 . We trained the model for a maximum of 20 epochs and used early stopping with the patience set to 5 epochs.

Results and Discussions
We now discuss the results of our experiments.

Comparison of different models
Following lebret2016neural, we used BLEU-4, NIST-4 and ROUGE-4 as the evaluation metrics. We first make a few observations based on the results on the English dataset (Table TABREF15 ). The basic seq2seq model, as well as the model proposed by weather16, perform better than the model proposed by lebret2016neural. Our final model with bifocal attention and gated orthogonalization gives the best performance and does 10% (relative) better than the closest baseline (basic seq2seq) and 21% (relative) better than the current state of the art method BIBREF0 . In Table TABREF16 , we show some qualitative examples of the output generated by different models.

Performance on different languages
The results on the French and German datasets are summarized in Tables TABREF20 and TABREF20 respectively. Note that the code of BIBREF0 is not publicly available, hence we could not report numbers for French and German using their model. We observe that our final model gives the best performance - though the bifocal attention model performs poorly as compared to the basic seq2seq model on French. However, the overall performance for French and German are much smaller than those for English. There could be multiple reasons for this. First, the amount of training data in these two languages is smaller than that in English. Specifically, the amount of training data available in French (German) is only INLINEFORM0 ( INLINEFORM1 )% of that available for English. Second, on average the descriptions in French and German are longer than that in English (EN: INLINEFORM2 words, FR: INLINEFORM3 words and DE: INLINEFORM4 words). Finally, a manual inspection across the three languages suggests that the English descriptions have a more consistent structure than the French descriptions. For example, most English descriptions start with name followed by date of birth but this is not the case in French. However, this is only a qualitative observation and it is hard to quantify this characteristic of the French and German datasets.

Visualizing Attention Weights
If the proposed model indeed works well then we should see attention weights that are consistent with the stay on and never look back behavior. To verify this, we plotted the attention weights in cases where the model with gated orthogonalization does better than the model with only bifocal attention. Figure FIGREF21 shows the attention weights corresponding to infobox in Figure FIGREF25 . Notice that the model without gated orthogonalization has attention on both name field and article title while rendering the name. The model with gated orthogonalization, on the other hand, stays on the name field for as long as it is required but then moves and never returns to it (as expected).
Due to lack of space, we do not show similar plots for French and German but we would like to mention that, in general, the differences between the attention weights learned by the model with and without gated orthogonalization were more pronounced for the French/German dataset than the English dataset. This is in agreement with the results reported in Table TABREF20 and TABREF20 where the improvements given by gated orthogonalization are more for French/German than for English.

Figure 2: Proposed model

Table 1: Comparison of different models on the English WIKIBIO dataset

Table 5: Comparison of different models on the German WIKIBIO dataset

Table 2: Examples of generated descriptions from different models. For the last two examples, name generated by Basic Seq2Seq model is incorrect because it attended to preceded by field.

Table 6: Out of domain results(BLEU-4)

Figure 4: Wikipedia Infobox for Samuel Smiles

Figure 5: Wikipedia Infobox for Mark Tobey

Figure 6: Comparison of the attention weights and descriptions (see highlighted boxes) produced by an out-of-domain model with and without fine tuning for the Infobox in Figure 5

Question: What metrics are used for evaluation?
{"answer": ["BLEU-4", "NIST-4", "ROUGE-4"], "evidence": ["Following lebret2016neural, we used BLEU-4, NIST-4 and ROUGE-4 as the evaluation metrics. We first make a few observations based on the results on the English dataset (Table TABREF15 ). The basic seq2seq model, as well as the model proposed by weather16, perform better than the model proposed by lebret2016neural. Our final model with bifocal attention and gated orthogonalization gives the best performance and does 10% (relative) better than the closest baseline (basic seq2seq) and 21% (relative) better than the current state of the art method BIBREF0 . In Table TABREF16 , we show some qualitative examples of the output generated by different models."]}