You will be shown sections from a scientific research paper, together with a question about the paper. Paragraphs in the paper are separated by newlines. Your task is to answer the question based on the contents of the paper.

Paper:
----------------------------------------
Entity Commonsense Representation for Neural Abstractive Summarization

A major proportion of a text summary includes important entities found in the original text. These entities build up the topic of the summary. Moreover, they hold commonsense information once they are linked to a knowledge base. Based on these observations, this paper investigates the usage of linked entities to guide the decoder of a neural text summarizer to generate concise and better summaries. To this end, we leverage on an off-the-shelf entity linking system (ELS) to extract linked entities and propose Entity2Topic (E2T), a module easily attachable to a sequence-to-sequence model that transforms a list of entities into a vector representation of the topic of the summary. Current available ELS's are still not sufficiently effective, possibly introducing unresolved ambiguities and irrelevant entities. We resolve the imperfections of the ELS by (a) encoding entities with selective disambiguation, and (b) pooling entity vectors using firm attention. By applying E2T to a simple sequence-to-sequence model with attention mechanism as base model, we see significant improvements of the performance in the Gigaword (sentence to title) and CNN (long document to multi-sentence highlights) summarization datasets by at least 2 ROUGE points.

Usefulness of linked entities in summarization
In the next subsections, we present detailed arguments with empirical and previously examined evidences on the observations and possible issues when using linked entities extracted by an entity linking system (ELS) for generating abstractive summaries. For this purpose, we use the development sets of the Gigaword dataset provided in BIBREF5 and of the CNN dataset provided in BIBREF6 as the experimental data for quantitative evidence and refer the readers to Figure 1 as the running example.

Observations
As discussed in Section "Introduction" , we find three observations that show the usefulness of linked entities for abstractive summarization.
First, summaries are mainly composed of linked entities extracted from the original text. In the example, it can be seen that the summary contains four words that refer to different entities. In fact, all noun phrases in the summary mention at least one linked entity. In our experimental data, we extract linked entities from the original text and compare them to the noun phrases found in the summary. We report that $77.1\%$ and $75.1\%$ of the noun phrases on the Gigaword and CNN datasets, respectively, contain at least one linked entity, which confirms our observation.
Second, linked entities can be used to represent the topic of the summary, defined as a multinomial distribution over entities, as graphically shown in the example, where the probabilities refer to the relevance of the entities. Entities have been previously used to represent topics BIBREF7 , as they can be utilized as a controlled vocabulary of the main topics in a document BIBREF8 . In the example, we see that the entity “Jae Seo” is the most relevant because it is the subject of the summary, while the entity “South Korean” is less relevant because it is less important when constructing the summary.
Third, we can make use of the entity commonsense that can be learned as a continuous vector representation from a separate larger corpus BIBREF9 , BIBREF10 . In the example, if we know that the entities “Los Angeles Dodgers” and “New York Mets” are American baseball teams and “Jae Seo” is a baseball player associated with the teams, then we can use this information to generate more coherent summaries. We find that $76.0\%$ of the extracted linked entities are covered by the pre-trained vectors in our experimental data, proving our third observation.

Possible issues
Despite its usefulness, linked entities extracted from ELS's have issues because of low precision rates BIBREF11 and design challenges in training datasets BIBREF12 . These issues can be summarized into two parts: ambiguity and coarseness.
First, the extracted entities may be ambiguous. In the example, the entity “South Korean” is ambiguous because it can refer to both the South Korean person and the South Korean language, among others. In our experimental data, we extract (1) the top 100 entities based on frequency, and (2) the entities extracted from 100 randomly selected texts, and check whether they have disambiguation pages in Wikipedia or not. We discover that $71.0\%$ of the top 100 entities and $53.6\%$ of the entities picked at random have disambiguation pages, which shows that most entities are prone to ambiguity problems.
Second, the linked entities may also be too common to be considered an entity. This may introduce errors and irrelevance to the summary. In the example, “Wednesday” is erroneous because it is wrongly linked to the entity “Wednesday Night Baseball”. Also, “swap” is irrelevant because although it is linked correctly to the entity “Trade (Sports)”, it is too common and irrelevant when generating the summaries. In our experimental data, we randomly select 100 data instances and tag the correctness and relevance of extracted entities into one of four labels: A: correct and relevant, B: correct and somewhat relevant, C: correct but irrelevant, and D: incorrect. Results show that $29.4\%$ , $13.7\%$ , $30.0\%$ , and $26.9\%$ are tagged with A, B, C, and D, respectively, which shows that there is a large amount of incorrect and irrelevant entities.

Our model
To solve the issues described above, we present Entity2Topic (E2T), a module that can be easily attached to any sequence-to-sequence based abstractive summarization model. E2T encodes the linked entities extracted from the text and transforms them into a single topic vector. This vector is ultimately concatenated to the decoder hidden state vectors. The module contains two submodules specifically for the issues presented by the entity linking systems: the entity encoding submodule with selective disambiguation and the pooling submodule with firm attention.
Overall, our full architecture can be illustrated as in Figure 2 , which consists of an entity linking system (ELS), a sequence-to-sequence with attention mechanism model, and the E2T module. We note that our proposed module can be easily attached to more sophisticated abstractive summarization models BIBREF13 , BIBREF14 that are based on the traditional encoder-decoder framework and consequently can produce better results. The code of the base model and the E2T are available online.

Pooling submodule
The entity vectors $\tilde{E}$ are pooled to create a single topic vector $t$ that represents the topic of the summary. One possible pooling technique is to use soft attention BIBREF18 on the vectors to determine the importance value of each vector, which can be done by matching each entity vector with the text vector $s$ from the text encoder as the context vector. The entity vectors are then pooled using weighted sum. One problem with soft attention is that it considers all entity vectors when constructing the topic vector. However, not all entities are important and necessary when generating summaries. Moreover, a number of these entities may be erroneous and irrelevant, as reported in Section "Related work" . Soft attention gives non-negligible important scores to these entities, thus adds unnecessary noise to the construction of the topic vector.
Our pooling submodule instead uses firm attention mechanism to consider only top $k$ entities when constructing the topic vector. This is done in a differentiable way as follows: $
G &= v_a^\top tanh(W_a \tilde{E} + U_a s) \\
K &= top\_k(G) \\
P &= sparse\_vector(K, 0, -\infty ) \\
g^{\prime }_i &= g_i + p_i \\
a_i &= \frac{exp(g^{\prime }_i)}{\sum _i exp(g^{\prime }_i)} \\
t &= \sum _i a_i \tilde{e}_i \nonumber $ 
 where the functions $K = top\_k(G)$ gets the indices of the top $k$ vectors in $G$ and $P = sparse\_vector(K,0,-\infty )$ creates a sparse vector where the values of $K$ is 0 and $-\infty $ otherwise. The sparse vector $P$ is added to the original importance score vector $G$ to create a new importance score vector. In this new vector, important scores of non-top $k$0 entities are $k$1 . When softmax is applied, this gives very small, negligible, and close-to-zero values to non-top $k$2 entities. The value $k$3 depends on the lengths of the input text and summary. Moreover, when $k$4 increases towards infinity, firm attention becomes soft attention. We decide $k$5 empirically (see Section "Experimental settings" ).

Extending from the base model
Entity2Topic module extends the base model as follows. The final text representation vector $s$ is used as a context vector when constructing the topic vector $t$ in the pooling submodule. The topic vector $t$ is then concatenated to the decoder hidden state vectors $s_i$ , i.e. $s^{\prime }_i = [s_i; t]$ . The concatenated vector is finally used to create the output vector: 
$$o_i = W_w w_{i-1} + W_c c_i + W_s s^{\prime }_i \nonumber $$   (Eq. 18) 

Results
We report the ROUGE F1 scores for both datasets of all the competing models using ROUGE F1 scores BIBREF27 . We report the results on the Gigaword and the CNN dataset in Table 2 and Table 3 , respectively. In Gigaword dataset where the texts are short, our best model achieves a comparable performance with the current state-of-the-art. In CNN dataset where the texts are longer, our best model outperforms all the previous models. We emphasize that E2T module is easily attachable to better models, and we expect E2T to improve their performance as well. Overall, E2T achieves a significant improvement over the baseline model base, with at least 2 ROUGE-1 points increase in the Gigaword dataset and 6 ROUGE-1 points increase in the CNN dataset. In fact, all variants of E2T gain improvements over the baseline, implying that leveraging on linked entities improves the performance of the summarizer. Among the model variants, the CNN-based encoder with selective disambiguation and firm attention performs the best.
Automatic evaluation on the Gigaword dataset shows that the CNN and RNN variants of base+E2T have similar performance. To break the tie between both models, we also conduct human evaluation on the Gigaword dataset. We instruct two annotators to read the input sentence and rank the competing summaries from first to last according to their relevance and fluency: (a) the original summary gold, and from models (b) base, (c) base+E2Tcnn, and (d) base+E2Trnn. We then compute (i) the proportion of every ranking of each model and (ii) the mean rank of each model. The results are reported in Table 4 . The model with the best mean rank is base+E2Tcnn, followed by gold, then by base+E2Trnn and base, respectively. We also perform ANOVA and post-hoc Tukey tests to show that the CNN variant is significantly ( $p<0.01$ ) better than the RNN variant and the base model. The RNN variant does not perform as well as the CNN variant, contrary to the automatic ROUGE evaluation above. Interestingly, the CNN variant produces better (but with no significant difference) summaries than the gold summaries. We posit that this is due to the fact that the article title does not correspond to the summary of the first sentence.

Conclusion
We proposed to leverage on linked entities to improve the performance of sequence-to-sequence models on neural abstractive summarization task. Linked entities are used to guide the decoding process based on the summary topic and commonsense learned from a knowledge base. We introduced Entity2Topic (E2T), a module that is easily attachable to any model using an encoder-decoder framework. E2T applies linked entities into the summarizer by encoding the entities with selective disambiguation and pooling them into one summary topic vector with firm attention mechanism. We showed that by applying E2T to a basic sequence-to-sequence model, we achieve significant improvements over the base model and consequently achieve a comparable performance with more complex summarization models.

Figure 1: Observations on linked entities in summaries. O1: Summaries are mainly composed of entities. O2: Entities can be used to represent the topic of the summary. O3: Entity commonsense learned from a large corpus can be used.

Figure 2: Full architecture of our proposed sequence-to-sequence model with Entity2Topic (E2T) module.

Figure 3: Entity encoding submodule with selective disambiguation applied to the entity 3©. The left figure represents the full submodule while the right figure represents the two choices of disambiguating encoders.

Table 2: Results on the Gigaword dataset using the fulllength F1 variants of ROUGE.

Table 3: Results on the CNN dataset using the fulllength F1 ROUGE metric.

Table 5: Examples from Gigaword and CNN datasets and corresponding summaries generated by competing models. The tagged part of text is marked bold and preceded with at sign (@). The red color fill represents the attention scores given to each entity. We only report the attention scores of entities in the Gigaword example for conciseness since there are 80 linked entities in the CNN example.
----------------------------------------

Question: Why are current ELS's not sufficiently effective?

To answer the question, format your response as a `json` object with two fields:

"answer": A string providing a succinct answer to the question, in your own words.
"evidence": An array of strings. Each entry should be a full paragraph from the paper. Together, the evidence should serve as a justification for the answer.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": "BERT and RoBERTa."
  "evidence": ["We compare our approach against two baselines. In Table 1, we compare against BERT. In Table 2, we compare against RoBERTa. Our findings indicate that our approach improves over both baeslines..."]
}

The "answer" field should be roughly 46 characters in length.

Do not include any text in your response other than the json. If the question is unanswerable given the provided excerpts, respond with the single word "null".

To repeat, the question is: Why are current ELS's not sufficiently effective?

Answer JSON object:
{"answer": "Linked entities may be ambiguous or too common", "evidence": ["Despite its usefulness, linked entities extracted from ELS\u0027s have issues because of low precision rates BIBREF11 and design challenges in training datasets BIBREF12 . These issues can be summarized into two parts: ambiguity and coarseness.", "First, the extracted entities may be ambiguous. In the example, the entity \u201cSouth Korean\u201d is ambiguous because it can refer to both the South Korean person and the South Korean language, among others. In our experimental data, we extract (1) the top 100 entities based on frequency, and (2) the entities extracted from 100 randomly selected texts, and check whether they have disambiguation pages in Wikipedia or not. We discover that $71.0\\%$ of the top 100 entities and $53.6\\%$ of the entities picked at random have disambiguation pages, which shows that most entities are prone to ambiguity problems.", "Second, the linked entities may also be too common to be considered an entity. This may introduce errors and irrelevance to the summary. In the example, \u201cWednesday\u201d is erroneous because it is wrongly linked to the entity \u201cWednesday Night Baseball\u201d. Also, \u201cswap\u201d is irrelevant because although it is linked correctly to the entity \u201cTrade (Sports)\u201d, it is too common and irrelevant when generating the summaries. In our experimental data, we randomly select 100 data instances and tag the correctness and relevance of extracted entities into one of four labels: A: correct and relevant, B: correct and somewhat relevant, C: correct but irrelevant, and D: incorrect. Results show that $29.4\\%$ , $13.7\\%$ , $30.0\\%$ , and $26.9\\%$ are tagged with A, B, C, and D, respectively, which shows that there is a large amount of incorrect and irrelevant entities."]}