You will be shown sections from a scientific research paper, together with a question about the paper. Your task is to answer the question based on the contents of the paper. Your response should be a `json` object with two fields:

"answer": An array of strings extracted directly from the paper which, collectively, answer the question.
"evidence": An array of strings. Each should be an excerpt from the paper, in which one or more of the extracted answers can be found.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": ["BERT", "RoBERTa"],
  "evidence": ["In our experiments, we compare the performance of our model against BERT and RoBERTa."]
}

Do not include any text in your response other than the json.

If the question is unanswerable given the provided excerpts, respond with the single word "null".

Paper: A Framework for Evaluation of Machine Reading Comprehension Gold Standards

Machine Reading Comprehension (MRC) is the task of answering a question over a paragraph of text. While neural MRC systems gain popularity and achieve noticeable performance, issues are being raised with the methodology used to establish their performance, particularly concerning the data design of gold standards that are used to evaluate them. There is but a limited understanding of the challenges present in this data, which makes it hard to draw comparisons and formulate reliable hypotheses. As a first step towards alleviating the problem, this paper proposes a unifying framework to systematically investigate the present linguistic features, required reasoning and background knowledge and factual correctness on one hand, and the presence of lexical cues as a lower bound for the requirement of understanding on the other hand. We propose a qualitative annotation schema for the first and a set of approximative metrics for the latter. In a first application of the framework, we analyse modern MRC gold standards and present our findings: the absence of features that contribute towards lexical ambiguity, the varying factual correctness of the expected answers and the presence of lexical cues, all of which potentially lower the reading comprehension complexity and quality of the evaluation data.

Framework for MRC Gold Standard Analysis ::: Problem definition
We define the task of machine reading comprehension, the target application of the proposed methodology as follows: Given a paragraph $P$ that consists of tokens (words) $p_1, \ldots , p_{n_P}$ and a question $Q$ that consists of tokens $q_1 \ldots q_{n_Q}$, the goal is to retrieve an answer $A$ with tokens $a_1 \ldots a_{n_A}$. $A$ is commonly constrained to be one of the following cases BIBREF15, illustrated in Figure FIGREF9:
Multiple choice, where the goal is to predict $A$ from a given set of choices $\mathcal {A}$.
Cloze-style, where $S$ is a sentence, and $A$ and $Q$ are obtained by removing a sequence of words such that $Q = S - A$. The task is to fill in the resulting gap in $Q$ with the expected answer $A$ to form $S$.
Span, where is a continuous subsequence of tokens from the paragraph ($A \subseteq P$). Flavours include multiple spans as the correct answer or $A \subseteq Q$.
Free form, where $A$ is an unconstrained natural language string.
A gold standard $G$ is composed of $m$ entries $(Q_i, A_i, P_i)_{i\in \lbrace 1,\ldots , m\rbrace }$.
The performance of an approach is established by comparing its answer predictions $A^*_{i}$ on the given input $(Q_i, T_i)$ (and $\mathcal {A}_i$ for the multiple choice setting) against the expected answer $A_i$ for all $i\in \lbrace 1,\ldots , m\rbrace $ under a performance metric. Typical performance metrics are exact match (EM) or accuracy, i.e. the percentage of exactly predicted answers, and the F1 score – the harmonic mean between the precision and the recall of the predicted tokens compared to expected answer tokens. The overall F1 score can either be computed by averaging the F1 scores for every instance or by first averaging the precision and recall and then computing the F1 score from those averages (macro F1). Free-text answers, meanwhile, are evaluated by means of text generation and summarisation metrics such as BLEU BIBREF16 or ROUGE-L BIBREF17.

Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Problem setting
We are interested in different types of the expected answer. We differentiate between Span, where an answer is a continuous span taken from the passage, Paraphrasing, where the answer is a paraphrase of a text span, Unanswerable, where there is no answer present in the context, and Generated, if it does not fall into any of the other categories. It is not sufficient for an answer to restate the question or combine multiple Span or Paraphrasing answers to be annotated as Generated. It is worth mentioning that we focus our investigations on answerable questions. For a complementary qualitative analysis that categorises unanswerable questions, the reader is referred to Yatskar2019.
Furthermore, we mark a sentence as Supporting Fact if it contains evidence required to produce the expected answer, as they are used further in the complexity analysis.

Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Factual Correctness
An important factor for the quality of a benchmark is its factual correctness, because on the one hand, the presence of factually wrong or debatable examples introduces an upper bound for the achievable performance of models on those gold standards. On the other hand, it is hard to draw conclusions about the correctness of answers produced by a model that is evaluated on partially incorrect data.
One way by which developers of modern crowd-sourced gold standards ensure quality is by having the same entry annotated by multiple workers BIBREF18 and keeping only those with high agreement. We investigate whether this method is enough to establish a sound ground truth answer that is unambiguously correct. Concretely we annotate an answer as Debatable when the passage features multiple plausible answers, when multiple expected answers contradict each other, or an answer is not specific enough with respect to the question and a more specific answer is present. We annotate an answer as Wrong when it is factually wrong and a correct answer is present in the context.

Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Required Reasoning
It is important to understand what types of reasoning the benchmark evaluates, in order to be able to accredit various reasoning capabilities to the models it evaluates. Our proposed reasoning categories are inspired by those found in scientific question answering literature BIBREF19, BIBREF20, as research in this area focuses on understanding the required reasoning capabilities. We include reasoning about the Temporal succession of events, Spatial reasoning about directions and environment, and Causal reasoning about the cause-effect relationship between events. We further annotate (multiple-choice) answers that can only be answered By Exclusion of every other alternative.
We further extend the reasoning categories by operational logic, similar to those required in semantic parsing tasks BIBREF21, as solving those tasks typically requires “multi-hop” reasoning BIBREF14, BIBREF22. When an answer can only be obtained by combining information from different sentences joined by mentioning a common entity, concept, date, fact or event (from here on called entity), we annotate it as Bridge. We further annotate the cases, when the answer is a concrete entity that satisfies a Constraint specified in the question, when it is required to draw a Comparison of multiple entities' properties or when the expected answer is an Intersection of their properties (e.g. “What do Person A and Person B have in common?”)
We are interested in the linguistic reasoning capabilities probed by a gold standard, therefore we include the appropriate category used by Wang2019. Specifically, we annotate occurrences that require understanding of Negation, Quantifiers (such as “every”, “some”, or “all”), Conditional (“if ...then”) statements and the logical implications of Con-/Disjunction (i.e. “and” and “or”) in order to derive the expected answer.
Finally, we investigate whether arithmetic reasoning requirements emerge in MRC gold standards as this can probe for reasoning that is not evaluated by simple answer retrieval BIBREF23. To this end, we annotate the presence of of Addition and Subtraction, answers that require Ordering of numerical values, Counting and Other occurrences of simple mathematical operations.
An example can exhibit multiple forms of reasoning. Notably, we do not annotate any of the categories mentioned above if the expected answer is directly stated in the passage. For example, if the question asks “How many total points were scored in the game?” and the passage contains a sentence similar to “The total score of the game was 51 points”, it does not require any reasoning, in which case we annotate it as Retrieval.

Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Linguistic Complexity
Another dimension of interest is the evaluation of various linguistic capabilities of MRC models BIBREF25, BIBREF26, BIBREF27. We aim to establish which linguistic phenomena are probed by gold standards and to which degree. To that end, we draw inspiration from the annotation schema used by Wang2019, and adapt it around lexical semantics and syntax.
More specifically, we annotate features that introduce variance between the supporting facts and the question. With regard to lexical semantics, we focus on the use of redundant words that do not alter the meaning of a sentence for the task of retrieving the expected answer (Redundancy), requirements on the understanding of words' semantic fields (Lexical Entailment) and the use of Synonyms and Paraphrases with respect to the question wording. Furthermore we annotate cases where supporting facts contain Abbreviations of concepts introduced in the question (and vice versa) and when a Dative case substitutes the use of a preposition (e.g. “I bought her a gift” vs “I bought a gift for her”). Regarding syntax, we annotate changes from passive to active Voice, the substitution of a Genitive case with a preposition (e.g. “of”) and changes from nominal to verbal style and vice versa (Nominalisation).
We recognise features that add ambiguity to the supporting facts, for example when information is only expressed implicitly by using an Ellipsis. As opposed to redundant words, we annotate Restrictivity and Factivity modifiers, words and phrases whose presence does change the meaning of a sentence with regard to the expected answer, and occurrences of intra- or inter-sentence Coreference in supporting facts (that is relevant to the question). Lastly, we mark ambiguous syntactic features, when their resolution is required in order to obtain the answer. Concretely, we mark argument collection with con- and disjunctions (Listing) and ambiguous Prepositions, Coordination Scope and Relative clauses/Adverbial phrases/Appositions.

Application of the Framework ::: Candidate Datasets
We select contemporary MRC benchmarks to represent all four commonly used problem definitions BIBREF15. In selecting relevant datasets, we do not consider those that are considered “solved”, i.e. where the state of the art performance surpasses human performance, as is the case with SQuAD BIBREF28, BIBREF7. Concretely, we selected gold standards that fit our problem definition and were published in the years 2016 to 2019, have at least $(2019 - publication\ year) \times 20$ citations, and bucket them according to the answer selection styles as described in Section SECREF4 We randomly draw one from each bucket and add two randomly drawn datasets from the candidate pool. This leaves us with the datasets described in Table TABREF19. For a more detailed description, we refer to Appendix .

Application of the Framework ::: Qualitative Analysis
We present a concise view of the annotation results in Figure FIGREF23. The full annotation results can be found in Appendix . We centre our discussion around the following main points:

Application of the Framework ::: Qualitative Analysis ::: Other
In the Figure FIGREF23 we observe that Operational and Arithmetic reasoning moderately (6% to 8% combined) appears “in the wild”, i.e. when not enforced by the data design as is the case with HotpotQA (80% Operations combined) or DROP (68% Arithmetic combined). Causal reasoning is (exclusively) present in MultiRC (32%), whereas Temporal and Spatial reasoning requirements seem to not naturally emerge in gold standards. In ReCoRd, a fraction of 38% questions can only be answered By Exclusion of every other candidate, due to the design choice of allowing questions where the required information to answer them is not fully expressed in the accompanying paragraph.
Therefore, it is also a little surprising to observe that ReCoRd requires external resources with regard to knowledge, as seen in Figure FIGREF23. MultiRC requires technical or more precisely basic scientific knowledge (6% Technical/Scientific), as a portion of paragraphs is extracted from elementary school science textbooks BIBREF31. Other benchmarks moderately probe for factual knowledge (0% to 4% across all categories), while Intuitive knowledge is required to derive answers in each gold standard.
It is also worth pointing out, as done in Figure FIGREF23, that although MultiRC and MsMarco are not modelled as a span selection problem, their samples still contain 50% and 66% of answers that are directly taken from the context. DROP contains the biggest fraction of generated answers (60%), due to the requirement of arithmetic operations.
To conclude our analysis, we observe similar distributions of linguistic features and reasoning patterns, except where there are constraints enforced by dataset design, annotation guidelines or source text choice. Furthermore, careful consideration of design choices (such as single-span answers) is required, to avoid impairing the factual correctness of datasets, as pure crowd-worker agreement seems not sufficient in multiple cases.

Related Work
Although not as prominent as the research on novel architecture, there has been steady progress in critically investigating the data and evaluation aspects of NLP and machine learning in general and MRC in particular.

Related Work ::: Adversarial Evaluation
The authors of the AddSent algorithm BIBREF11 show that MRC models trained and evaluated on the SQuAD dataset pay too little attention to details that might change the semantics of a sentence, and propose a crowd-sourcing based method to generate adversary examples to exploit that weakness. This method was further adapted to be fully automated BIBREF36 and applied to different gold standards BIBREF35. Our proposed approach differs in that we aim to provide qualitative justifications for those quantitatively measured issues.

Related Work ::: Sanity Baselines
Another line of research establishes sane baselines to provide more meaningful context to the raw performance scores of evaluated models. When removing integral parts of the task formulation such as question, the textual passage or parts thereof BIBREF37 or restricting model complexity by design in order to suppress some required form of reasoning BIBREF38, models are still able to perform comparably to the state-of-the-art. This raises concerns about the perceived benchmark complexity and is related to our work in a broader sense as one of our goals is to estimate the complexity of benchmarks.

Table 1: Summary of selected datasets

Table 2: Inter-Annotator agreement F1 scores, averaged for each dataset

Figure 4: Annotation results

Figure 5: Example of semantics altering lexical features

Table 7: Detailed results for the annotation of factual correctness. We calculate percentages relative to the number of examples that were annotated to be not unanswerable.

Table 9: Detailed linguistic feature results. We calculate percentages relative to the number of examples that were annotated to contain supporting facts.

Question: What modern MRC gold standards are analyzed?
{"answer": ["fit our problem definition and were published in the years 2016 to 2019, have at least $(2019 - publication\\ year) \\times 20$ citations"], "evidence": ["We select contemporary MRC benchmarks to represent all four commonly used problem definitions BIBREF15. In selecting relevant datasets, we do not consider those that are considered \u201csolved\u201d, i.e. where the state of the art performance surpasses human performance, as is the case with SQuAD BIBREF28, BIBREF7. Concretely, we selected gold standards that fit our problem definition and were published in the years 2016 to 2019, have at least $(2019 - publication\\ year) \\times 20$ citations, and bucket them according to the answer selection styles as described in Section SECREF4 We randomly draw one from each bucket and add two randomly drawn datasets from the candidate pool. This leaves us with the datasets described in Table TABREF19. For a more detailed description, we refer to Appendix ."]}