You will be shown sections from a scientific research paper, together with a question about the paper. Your task is to answer the question based on the contents of the paper. Your response should be a `json` object with two fields:

"answer": An array of strings extracted directly from the paper which, collectively, answer the question.
"evidence": An array of strings. Each should be an excerpt from the paper, in which one or more of the extracted answers can be found.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": ["BERT", "RoBERTa"],
  "evidence": ["In our experiments, we compare the performance of our model against BERT and RoBERTa."]
}

Do not include any text in your response other than the json.

If the question is unanswerable given the provided excerpts, respond with the single word "null".

Paper: Entity Commonsense Representation for Neural Abstractive Summarization

A major proportion of a text summary includes important entities found in the original text. These entities build up the topic of the summary. Moreover, they hold commonsense information once they are linked to a knowledge base. Based on these observations, this paper investigates the usage of linked entities to guide the decoder of a neural text summarizer to generate concise and better summaries. To this end, we leverage on an off-the-shelf entity linking system (ELS) to extract linked entities and propose Entity2Topic (E2T), a module easily attachable to a sequence-to-sequence model that transforms a list of entities into a vector representation of the topic of the summary. Current available ELS's are still not sufficiently effective, possibly introducing unresolved ambiguities and irrelevant entities. We resolve the imperfections of the ELS by (a) encoding entities with selective disambiguation, and (b) pooling entity vectors using firm attention. By applying E2T to a simple sequence-to-sequence model with attention mechanism as base model, we see significant improvements of the performance in the Gigaword (sentence to title) and CNN (long document to multi-sentence highlights) summarization datasets by at least 2 ROUGE points.

Usefulness of linked entities in summarization
In the next subsections, we present detailed arguments with empirical and previously examined evidences on the observations and possible issues when using linked entities extracted by an entity linking system (ELS) for generating abstractive summaries. For this purpose, we use the development sets of the Gigaword dataset provided in BIBREF5 and of the CNN dataset provided in BIBREF6 as the experimental data for quantitative evidence and refer the readers to Figure 1 as the running example.

Observations
As discussed in Section "Introduction" , we find three observations that show the usefulness of linked entities for abstractive summarization.
First, summaries are mainly composed of linked entities extracted from the original text. In the example, it can be seen that the summary contains four words that refer to different entities. In fact, all noun phrases in the summary mention at least one linked entity. In our experimental data, we extract linked entities from the original text and compare them to the noun phrases found in the summary. We report that $77.1\%$ and $75.1\%$ of the noun phrases on the Gigaword and CNN datasets, respectively, contain at least one linked entity, which confirms our observation.
Second, linked entities can be used to represent the topic of the summary, defined as a multinomial distribution over entities, as graphically shown in the example, where the probabilities refer to the relevance of the entities. Entities have been previously used to represent topics BIBREF7 , as they can be utilized as a controlled vocabulary of the main topics in a document BIBREF8 . In the example, we see that the entity “Jae Seo” is the most relevant because it is the subject of the summary, while the entity “South Korean” is less relevant because it is less important when constructing the summary.
Third, we can make use of the entity commonsense that can be learned as a continuous vector representation from a separate larger corpus BIBREF9 , BIBREF10 . In the example, if we know that the entities “Los Angeles Dodgers” and “New York Mets” are American baseball teams and “Jae Seo” is a baseball player associated with the teams, then we can use this information to generate more coherent summaries. We find that $76.0\%$ of the extracted linked entities are covered by the pre-trained vectors in our experimental data, proving our third observation.

Possible issues
Despite its usefulness, linked entities extracted from ELS's have issues because of low precision rates BIBREF11 and design challenges in training datasets BIBREF12 . These issues can be summarized into two parts: ambiguity and coarseness.
First, the extracted entities may be ambiguous. In the example, the entity “South Korean” is ambiguous because it can refer to both the South Korean person and the South Korean language, among others. In our experimental data, we extract (1) the top 100 entities based on frequency, and (2) the entities extracted from 100 randomly selected texts, and check whether they have disambiguation pages in Wikipedia or not. We discover that $71.0\%$ of the top 100 entities and $53.6\%$ of the entities picked at random have disambiguation pages, which shows that most entities are prone to ambiguity problems.
Second, the linked entities may also be too common to be considered an entity. This may introduce errors and irrelevance to the summary. In the example, “Wednesday” is erroneous because it is wrongly linked to the entity “Wednesday Night Baseball”. Also, “swap” is irrelevant because although it is linked correctly to the entity “Trade (Sports)”, it is too common and irrelevant when generating the summaries. In our experimental data, we randomly select 100 data instances and tag the correctness and relevance of extracted entities into one of four labels: A: correct and relevant, B: correct and somewhat relevant, C: correct but irrelevant, and D: incorrect. Results show that $29.4\%$ , $13.7\%$ , $30.0\%$ , and $26.9\%$ are tagged with A, B, C, and D, respectively, which shows that there is a large amount of incorrect and irrelevant entities.

Base model
As our base model, we employ a basic encoder-decoder RNN used in most neural machine translation BIBREF2 and text summarization BIBREF15 tasks. We employ a two-layer bidirectional GRU (BiGRU) as the recurrent unit of the encoder. The BiGRU consists of a forward and backward GRU, which results to sequences of forward and backward hidden states $(\overrightarrow{h}_1, \overrightarrow{h}_2, ..., \overrightarrow{h}_n)$ and $(\overleftarrow{h}_1, \overleftarrow{h}_2, ..., \overleftarrow{h}_n)$ , respectively: $
\overrightarrow{h}_i &= GRU(x_i, \overrightarrow{h}_{i-1}) \\
\overleftarrow{h}_i &= GRU(x_i, \overleftarrow{h}_{i+1}) \nonumber $ 
The forward and backward hidden states are concatenated to get the hidden state vectors of the tokens (i.e. $h_i = [\overrightarrow{h}_i; \overleftarrow{h}_i]$ ). The final states of the forward and backward GRU are also concatenated to create the final text representation vector of the encoder $s = [\overrightarrow{h}_n; \overleftarrow{h}_1]$ . These values are calculated per layer, where $x_t$ of the second layer is $h_t$ of the first layer. The final text representation vectors are projected by a fully connected layer and are passed to the decoder as the initial hidden states $s_0 = s$ .
For the decoder, we use a two-layer uni-directional GRU with attention. At each time step $t$ , the previous token $y_{t-1}$ , the previous hidden state $s_{t-1}$ , and the previous context vector $c_{t-1}$ are passed to a GRU to calculate the new hidden state $s_t$ , as shown in the equation below. 
$$s_t = GRU(w_{t-1}, s_{t-1}, c_{t-1}) \nonumber $$   (Eq. 9) 
The context vector $c_t$ is computed using the additive attention mechanism BIBREF2 , which matches the current decoder state $s_t$ and each encoder state $h_i$ to get an importance score. The scores are then passed to a softmax and are used to pool the encoder states using weighted sum. The final pooled vector is the context vector, as shown in the equations below. $
g_{t,i} &= v_a^\top tanh(W_a s_{t-1} + U_a h_i) \\
a_{t,i} &= \frac{exp(g_{t,i})}{\sum _i exp(g_{t,i})} \\
c_t &= \sum _i a_{t,i} h_i \nonumber $ 
Finally, the previous token $y_{t-1}$ , the current context vector $c_t$ , and the current decoder state $s_t$ are used to generate the current word $y_t$ with a softmax layer over the decoder vocabulary, as shown below. $
o_t &= W_w w_{t-1} + W_c c_t + W_s s_t \\
p(y_t | y_{<t}) &= softmax(W_o o_t) \nonumber $ 

Entity encoding submodule
After performing entity linking to the input text using the ELS, we receive a sequential list of linked entities, arranged based on their location in the text. We embed these entities to $d$ -dimensional vectors $E = \lbrace e_1, e_2, ..., e_m\rbrace $ where $e_i \in \mathbb {R}^d$ . Since these entities may still contain ambiguity, it is necessary to resolve them before applying them to the base model. Based on the idea that an ambiguous entity can be disambiguated using its neighboring entities, we introduce two kinds of disambiguating encoders below.
One way to disambiguate an entity is by using all the other entities, putting more importance to entities that are nearer. For this purpose, we employ an RNN-based model to globally disambiguate the entities. Specifically, we use BiGRU and concatenate the forward and backward hidden state vectors as the new entity vector: $
\overrightarrow{h}_i &= GRU(e_i, \overrightarrow{h}_{i-1}) \\
\overleftarrow{h}_i &= GRU(e_i, \overleftarrow{h}_{i+1}) \\
e^{\prime }_i &= [\overrightarrow{h}_i; \overleftarrow{h}_i] \nonumber $ 
Another way to disambiguate an entity is by using only the direct neighbors of the entity, putting no importance value to entities that are far. To do this, we employ a CNN-based model to locally disambiguate the entities. Specifically, we do the convolution operation using filter matrices $W_f \in \mathbb {R}^{h \times d}$ with filter size $h$ to a window of $h$ words. We do this for different sizes of $h$ . This produces new feature vectors $c_{i,h}$ as shown below, where $f(.)$ is a non-linear function: $
c_{i,h} = f([e_{i-(h-1)/2}; ...; e_{i+h(+1)/2}]^\top W_f + b_f) \nonumber $ 
The convolution operation reduces the number of entities differently depending on the filter size $h$ . To prevent loss of information and to produce the same amount of feature vectors $c_{i,h}$ , we pad the entity list dynamically such that when the filter size is $h$ , the number of paddings on each side is $(h-1)/2$ . The filter size $h$ therefore refers to the number of entities used to disambiguate a middle entity. Finally, we concatenate all feature vectors of different $h$ 's for each $i$ as the new entity vector: $
e^{\prime }_i = [c_{i,h_1}; c_{i, h_2}; ...] \nonumber $ 
The question on which disambiguating encoder is better has been a debate; some argued that using only the local context is appropriate BIBREF16 while some claimed that additionally using global context also helps BIBREF17 . The RNN-based encoder is good as it smartly makes use of all entities, however it may perform bad when there are many entities as it introduces noise when using a far entity during disambiguation. The CNN-based encoder is good as it minimizes the noise by totally ignoring far entities when disambiguating, however determining the appropriate filter sizes $h$ needs engineering. Overall, we argue that when the input text is short (e.g. a sentence), both encoders perform comparably, otherwise when the input text is long (e.g. a document), the CNN-based encoder performs better.
It is obvious that not all entities need to be disambiguated. When a correctly linked and already adequately disambiguated entity is disambiguated again, it would make the entity very context-specific and might not be suitable for the summarization task. Our entity encoding submodule therefore uses a selective mechanism that decides whether to use the disambiguating encoder or not. This is done by introducing a selective disambiguation gate $d$ . The final entity vector $\tilde{e}_i$ is calculated as the linear transformation of $e_i$ and $e^{\prime }_i$ : $
e^{\prime }_i &= encoder(e_i) \\
d &= \sigma (W_d e^{\prime }_i + b_d) \\
\tilde{e}_i &= d \times f(W_x e_i + b_x) + \\ & \quad (1-d) \times f(W_y e^{\prime }_i + b_y) \nonumber $ 
The full entity encoding submodule is illustrated in Figure 3 . Ultimately, the submodule outputs the disambiguated entity vectors $\tilde{E} = \lbrace \tilde{e}_1, \tilde{e}_2, ..., \tilde{e}_m\rbrace $ .

Figure 2: Full architecture of our proposed sequence-to-sequence model with Entity2Topic (E2T) module.

Figure 3: Entity encoding submodule with selective disambiguation applied to the entity 3©. The left figure represents the full submodule while the right figure represents the two choices of disambiguating encoders.

Table 3: Results on the CNN dataset using the fulllength F1 ROUGE metric.

Table 4: Human evaluations on the Gigaword dataset. Bold-faced values are the best while red-colored values are the worst among the values in the evaluation metric.

Table 6: Examples with highest/lowest disambiguation gate d values of two example entities (United States and gold). The tagged part of text is marked bold and preceded with at sign (@).

Question: Why are current ELS's not sufficiently effective?
{"answer": ["linked entities extracted from ELS\u0027s have issues because of low precision rates BIBREF11 and design challenges in training datasets BIBREF12 . These issues can be summarized into two parts: ambiguity and coarseness.", "the linked entities may also be too common to be considered an entity."], "evidence": ["Despite its usefulness, linked entities extracted from ELS\u0027s have issues because of low precision rates BIBREF11 and design challenges in training datasets BIBREF12 . These issues can be summarized into two parts: ambiguity and coarseness.", "Second, the linked entities may also be too common to be considered an entity. This may introduce errors and irrelevance to the summary. In the example, \u201cWednesday\u201d is erroneous because it is wrongly linked to the entity \u201cWednesday Night Baseball\u201d. Also, \u201cswap\u201d is irrelevant because although it is linked correctly to the entity \u201cTrade (Sports)\u201d, it is too common and irrelevant when generating the summaries. In our experimental data, we randomly select 100 data instances and tag the correctness and relevance of extracted entities into one of four labels: A: correct and relevant, B: correct and somewhat relevant, C: correct but irrelevant, and D: incorrect. Results show that $29.4\\%$ , $13.7\\%$ , $30.0\\%$ , and $26.9\\%$ are tagged with A, B, C, and D, respectively, which shows that there is a large amount of incorrect and irrelevant entities."]}