You will be shown sections from a scientific research paper, together with a question about the paper. Your task is to answer the question based on the contents of the paper. Your response should be a `json` object with two fields:

"answer": An array of strings extracted directly from the paper which, collectively, answer the question.
"evidence": An array of strings. Each should be an excerpt from the paper, in which one or more of the extracted answers can be found.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": ["BERT", "RoBERTa"],
  "evidence": ["In our experiments, we compare the performance of our model against BERT and RoBERTa."]
}

Do not include any text in your response other than the json.

If the question is unanswerable given the provided excerpts, respond with the single word "null".

Paper: Embracing data abundance: BookTest Dataset for Reading Comprehension

There is a practically unlimited amount of natural language data available. Still, recent work in text comprehension has focused on datasets which are small relative to current computing possibilities. This article is making a case for the community to move to larger data and as a step in that direction it is proposing the BookTest, a new dataset similar to the popular Children's Book Test (CBT), however more than 60 times larger. We show that training on the new data improves the accuracy of our Attention-Sum Reader model on the original CBT test data by a much larger margin than many recent attempts to improve the model architecture. On one version of the dataset our ensemble even exceeds the human baseline provided by Facebook. We then show in our own human study that there is still space for further improvement.

Task Description
A natural way of testing a reader's comprehension of a text is to ask her a question the answer to which can be deduced from the text. Hence the task we are trying to solve consists of answering a cloze-style question, the answer to which depends on the understanding of a context document provided with the question. The model is also provided with a set of possible answers from which the correct one is to be selected. This can be formalized as follows:
The training data consist of tuples INLINEFORM0 , where INLINEFORM1 is a question, INLINEFORM2 is a document that contains the answer to question INLINEFORM3 , INLINEFORM4 is a set of possible answers and INLINEFORM5 is the ground-truth answer. Both INLINEFORM6 and INLINEFORM7 are sequences of words from vocabulary INLINEFORM8 . We also assume that all possible answers are words from the vocabulary, that is INLINEFORM9 . In the CBT and CNN/Daily Mail datasets it is also true that the ground-truth answer INLINEFORM10 appears in the document. This is exploited by many machine learning models BIBREF2 , BIBREF4 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF10 , BIBREF11 , BIBREF12 , however some do not explicitly depend on this property BIBREF1 , BIBREF3 , BIBREF5 , BIBREF9 

Current Landscape
We will now briefly review what datasets for text comprehension have been published up to date and look at models which have been recently applied to solving the task we have just described.

Datasets
A crucial condition for applying deep-learning techniques is to have a huge amount of data available for training. For question answering this specifically means having a large number of document-question-answer triples available. While there is an unlimited amount of text available, coming up with relevant questions and the corresponding answers can be extremely labour-intensive if done by human annotators. There were efforts to provide such human-generated datasets, e.g. Microsoft's MCTest BIBREF17 , however their scale is not suitable for deep learning without pre-training on other data BIBREF18 (such as using pre-trained word embedding vectors).
Google DeepMind managed to avoid this scale issue with their way of generating document-question-answer triples automatically, closely followed by Facebook with a similar method. Let us now briefly introduce the two resulting datasets whose properties are summarized in Table TABREF8 .
These two datasets BIBREF1 exploit a useful feature of online news articles – many articles include a short summarizing sentence near the top of the page. Since all information in the summary sentence is also presented in the article body, we get a nice cloze-style question about the article contents by removing a word from the short summary.
The dataset's authors also replaced all named entities in the dataset by anonymous tokens which are further shuffled for each new batch. This forces the model to rely solely on information from the context document, not being able to transfer any meaning of the named entities between documents.
This restricts the task to one specific aspect of context-dependent question answering which may be useful however it moves the task further from the real application scenario, where we would like the model to use all information available to answer questions. Furthermore Chen et al. BIBREF5 have suggested that this can make about 17% of the questions unanswerable even by humans. They also claim that more than a half of the question sentences are mere paraphrases or exact matches of a single sentence from the context document. This raises a question to what extent the dataset can test deeper understanding of the articles.
The Children's Book Test BIBREF2 uses a different source - books freely available thanks to Project Gutenberg. Since no summary is available, each example consists of a context document formed from 20 consecutive sentences from the story together with a question formed from the subsequent sentence.
The dataset comes in four flavours depending on what type of word is omitted from the question sentence. Based on human evaluation done in BIBREF2 it seems that NE and CN are more context dependent than the other two types – prepositions and verbs. Therefore we (and all of the recent publications) focus only on these two word types.
Several new datasets related to the (now almost standard) ones above emerged recently. We will now briefly present them and explain how the dataset we are introducing in this article differs from them.
The LAMBADA dataset BIBREF19 is designed to measure progress in understanding common-sense questions about short stories that can be easily answered by humans but cannot be answered by current standard machine-learning models (e.g. plain LSTM language models). This dataset is useful for measuring the gap between humans and machine learning algorithms. However, by contrast to our BookTest dataset, it will not allow us to track progress towards the performance of the baseline systems or on examples where machine learning may show super-human performance. Also LAMBADA is just a diagnostic dataset and does not provide ready-to-use question-answering training data, just a plain-text corpus which may moreover include copyrighted books making its use potentially problematic for some purposes. We are providing ready training data consisting of copyright-free books only.
The SQuAD dataset BIBREF20 based on Wikipedia and the Who-did-What dataset BIBREF21 based on Gigaword news articles are factoid question-answering datasets where a multi-word answer should be extracted from a context document. This is in contrast to the previous datasets, including CNN/DM, CBT, LAMBADA and our new dataset, which require only single-word answers. Both these datasets however provide less than 130,000 training questions, two orders of magnitude less than our dataset does.
The Story Cloze Test BIBREF22 provides a crowd-sourced corpus of 49,255 commonsense stories for training and 3,744 testing stories with right and wrong endings. Hence the dataset is again rather small. Similarly to LAMBADA, the Story Cloze Test was designed to be easily answerable by humans.
In the WikiReading BIBREF23 dataset the context document is formed from a Wikipedia article and the question-answer pair is taken from the corresponding WikiData page. For each entity (e.g. Hillary Clinton), WikiData contain a number of property-value pairs (e.g. place of birth: Chicago) which form the datasets's question-answer pairs. The dataset is certainly relevant to the community, however the questions are of very limited variety with only 20 properties (and hence unique questions) covering INLINEFORM0 of the dataset. Furthermore many of the frequent properties are mentioned at a set spot within the article (e.g. the date of birth is almost always in brackets behind the name of a person) which may make the task easier for machines. We are trying to provide a more varied dataset.
Although there are several datasets related to task we are aiming to solve, they differ sufficiently for our dataset to bring new value to the community. Its biggest advantage is its size which can furthermore be easily upscaled without expensive human annotation. Finally while we are emphasizing the differences, models could certainly benefit from as diverse a collection of datasets as possible.

Machine Learning Models
A first major work applying deep-learning techniques to text comprehension was Hermann et al. BIBREF1 . This work was followed by the application of Memory Networks to the same task BIBREF2 . Later three models emerged around the same time BIBREF3 , BIBREF4 , BIBREF5 including our psr model BIBREF4 . The AS Reader inspired several subsequent models that use it as a sub-component in a diverse ensemble BIBREF8 ; extend it with a hierarchical structure BIBREF6 , BIBREF24 , BIBREF7 ; compute attention over the context document for every word in the query BIBREF10 or use two-way context-query attention mechanism for every word in the context and the query BIBREF11 that is similar in its spirit to models recently proposed in different domains, e.g. BIBREF25 in information retrieval. Other neural approaches to text comprehension are explored in BIBREF9 , BIBREF12 .

Baselines
We will now use our psr model to evaluate the performance gain from increasing the dataset size.

AS Reader
In BIBREF4 we introduced the psr , which at the time of publication significantly outperformed all other architectures on the CNN, DM and CBT datasets. This model is built to leverage the fact that the answer is a single word from the context document. Similarly to many other models it uses attention over the document – intuitively a measure of how relevant each word is to answering the question. However while most previous models used this attention as weights to calculate a blended representation of the answer word, we simply sum the attention across all occurrences of each unique words and then simply select the word with the highest sum as the final answer. While simple, this trick seems both to improve accuracy and to speed-up training. It was adopted by many subsequent models BIBREF8 , BIBREF6 , BIBREF7 , BIBREF10 , BIBREF11 , BIBREF24 .
Let us now describe the model in more detail. Figure FIGREF21 may help you in understanding the following paragraphs.
The words from the document and the question are first converted into vector embeddings using a look-up matrix INLINEFORM0 . The document is then read by a bidirectional GRU network BIBREF29 . A concatenation of the hidden states of the forward and backward GRUs at each word is then used as a contextual embedding of this word, intuitively representing the context in which the word is appearing. We can also understand it as representing the set of questions to which this word may be an answer.
Similarly the question is read by a bidirectional GRU but in this case only the final hidden states are concatenated to form the question embedding.
The attention over each word in the context is then calculated as the dot product of its contextual embedding with the question embedding. This attention is then normalized by the softmax function and summed across all occurrences of each answer candidate. The candidate with most accumulated attention is selected as the final answer.
For a more detailed description of the model including equations check BIBREF4 . More details about the training setup and model hyperparameters can be found in the Appendix.
During our past experiments on the CNN, DM and CBT datasets BIBREF4 each unique word from the training, validation and test datasets had its row in the look-up matrix INLINEFORM0 . However as we radically increased the dataset size, this would result in an extremely large number of model parameters so we decided to limit the vocabulary size to INLINEFORM1 most frequent words. For each example, each unique out-of-vocabulary word is now mapped on one of 1000 anonymous tokens which are randomly initialized and untrained. Fixing the embeddings of these anonymous tags proved to significantly improve the performance.
While mostly using the original AS Reader model, we have also tried introducing a minor tweak in some instances of the model. We tried initializing the context encoder GRU's hidden state by letting the encoder read the question first before proceeding to read the context document. Intuitively this allows the encoder to know in advance what to look for when reading over the context document.
Including models of this kind in the ensemble helped to improve the performance.

Human Study
After adding more data we have the performance on the CBT validation and test datasets soaring. However is there still potential for much further growth beyond the results we have observed?
We decided to explore the remaining space for improvement on the CBT by testing humans on a random subset of 50 named entity and 50 common noun validation questions that the psr ensemble could not answer correctly. These questions were answered by 10 non-native English speakers from our research laboratory, each on a disjoint subset of questions.. Participants had unlimited time to answer the questions and were told that these questions were not correctly answered by a machine, providing additional motivation to prove they are better than computers. The results of the human study are summarized in Table TABREF28 . They show that a majority of questions that our system could not answer so far are in fact answerable. This suggests that 1) the original human baselines might have been underestimated, however, it might also be the case that there are some examples that can be answered by machines and not by humans; 2) there is still space for improvement.
A system that would answer correctly every time when either our ensemble or human answered correctly would achieve accuracy over 92% percent on both validation and test NE datasets and over 96% on both CN datasets. Hence it still makes sense to use CBT dataset to study further improvements of text-comprehension systems.

Training Details
The training details are similar to those in BIBREF4 however we are including them here for completeness.
To train the model we used stochastic gradient descent with the ADAM update rule BIBREF32 and learning rates of INLINEFORM0 , INLINEFORM1 and INLINEFORM2 . The best learning rate in our experiments was INLINEFORM3 . We minimized negative log-likelihood as the training objective.
The initial weights in the word-embedding matrix were drawn randomly uniformly from the interval INLINEFORM0 . Weights in the GRU networks were initialized by random orthogonal matrices BIBREF34 and biases were initialized to zero. We also used a gradient clipping BIBREF33 threshold of 10 and batches of sizes between 32 or 256. Increasing the batch from 32 to 128 seems to significantly improve performance on the large dataset - something we did not observe on the original CBT data. Increasing the batch size much above 128 is currently difficult due to memory constraints of the GPU.
During training we randomly shuffled all examples at the beginning of each epoch. To speed up training, we always pre-fetched 10 batches worth of examples and sorted them according to document length. Hence each batch contained documents of roughly the same length.
We also did not use pre-trained word embeddings.
We did not perform any text pre-processing since the datasets were already tokenized.
During training we evaluated the model performance every 12 hours and at the end of each epoch and stopped training when the error on the 20k BookTest validation set started increasing. We explored the hyperparameter space by training 67 different models The region of the parameter space that we explored together with the parameters of the model with best validation accuracy are summarized in Table TABREF29 .
Our model was implemented using Theano BIBREF31 and Blocks BIBREF35 .
The ensembles were formed by simply averaging the predictions from the constituent single models. These single models were selected using the following algorithm.
We started with the best performing model according to validation performance. Then in each step we tried adding the best performing model that had not been previously tried. We kept it in the ensemble if it did improve its validation performance and discarded it otherwise. This way we gradually tried each model once. We call the resulting model a greedy ensemble. We used the INLINEFORM0 BookTest validation dataset for this procedure.
The algorithm was offered 10 models and selected 5 of them for the final ensemble.

Figure 1: Structure of the AS Reader model.

Table 3: Accuracy of human participants on examples from validation set that were previously incorrectly answered by our system trained only on the BookTest data.

Table 4: The best hyperparameters and range of values that we tested on the BookTest train dataset. We report number of hidden units of the unidirectional GRU; the bidirectional GRU has twice as many hidden units.

Question: How does their ensemble method work?
{"answer": ["simply averaging the predictions from the constituent single models"], "evidence": ["The ensembles were formed by simply averaging the predictions from the constituent single models. These single models were selected using the following algorithm."]}