You will be shown sections from a scientific research paper, together with a question about the paper. Paragraphs in the paper are separated by newlines. Your task is to answer the question based on the contents of the paper.

Paper:
----------------------------------------
Generating Diverse Numbers of Diverse Keyphrases

Existing keyphrase generation studies suffer from the problems of generating duplicate phrases and deficient evaluation based on a fixed number of predicted phrases. We propose a recurrent generative model that generates multiple keyphrases sequentially from a text, with specific modules that promote generation diversity. We further propose two new metrics that consider a variable number of phrases. With both existing and proposed evaluation setups, our model demonstrates superior performance to baselines on three types of keyphrase generation datasets, including two newly introduced in this work: StackExchange and TextWorld ACG. In contrast to previous keyphrase generation approaches, our model generates sets of diverse keyphrases of a variable number.

Introduction
Keyphrase generation is the task of automatically predicting keyphrases given a source text. Desired keyphrases are often multi-word units that summarize the high-level meaning and highlight certain important topics or information of the source text. Consequently, models that can successfully perform this task should be capable of not only distilling high-level information from a document, but also locating specific, important snippets therein.
To make the problem even more challenging, a keyphrase may or may not be a substring of the source text (i.e., it may be present or absent). Moreover, a given source text is usually associated with a set of multiple keyphrases. Thus, keyphrase generation is an instance of the set generation problem, where both the size of the set and the size (i.e., the number of tokens in a phrase) of each element can vary depending on the source.
Similar to summarization, keyphrase generation is often formulated as a sequence-to-sequence (Seq2Seq) generation task in most prior studies BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 . Conditioned on a source text, Seq2Seq models generate phrases individually or as a longer sequence jointed by delimiting tokens. Since standard Seq2Seq models generate only one sequence at a time, thus to generate multiple phrases, a common approach is to over-generate using beam search with a large beam width. Models are then evaluated by taking a fixed number of top predicted phrases (typically 5 or 10) and comparing them against the ground truth keyphrases.
Though this approach has achieved good empirical results, we argue that it suffers from two major limitations. Firstly, models that use beam search to generate multiple keyphrases generally lack the ability to determine the dynamic number of keyphrases needed for different source texts. Meanwhile, the parallelism in beam search also fails to model the inter-relation among the generated phrases, which can often result in diminished diversity in the output. Although certain existing models take output diversity into consideration during training BIBREF1 , BIBREF2 , the effort is significantly undermined during decoding due to the reliance on over-generation and phrase ranking with beam search.
Secondly, the current evaluation setup is rather problematic, since existing studies attempt to match a fixed number of outputs against a variable number of ground truth keyphrases. Empirically, the number of keyphrases can vary drastically for different source texts, depending on a plethora of factors including the length or genre of the text, the granularity of keyphrase annotation, etc. For the several commonly used keyphrase generation datasets, for example, the average number of keyphrases per data point can range from 5.3 to 15.7, with variances sometimes as large as 64.6 (Table TABREF1 ). Therefore, using an arbitrary, fixed number INLINEFORM0 to evaluate entire datasets is not appropriate. In fact, under this evaluation setup, the F1 score for the oracle model on the KP20k dataset is 0.858 for INLINEFORM1 and 0.626 for INLINEFORM2 , which apparently poses serious normalization issues as evaluation metrics.
To overcome these problems, we propose novel decoding strategies and evaluation metrics for the keyphrase generation task. The main contributions of this work are as follows:

Decoding Strategies
According to different task requirements, various decoding methods can be applied to generate the target sequence INLINEFORM0 . Prior studies BIBREF0 , BIBREF7 focus more on generating excessive number of phrases by leveraging beam search to proliferate the output phrases. In contrast, models trained under One2Seq paradigm are capable of determining the proper number of phrases to output. In light of previous research in psychology BIBREF33 , BIBREF34 , we name these two decoding/search strategies as Exhaustive Decoding and Self-terminating Decoding, respectively, due to their resemblance to the way humans behave in serial memory tasks. Simply speaking, the major difference lies in whether a model is capable of controlling the number of phrases to output. We describe the detailed decoding strategies used in this study as follows:
As traditional keyphrase tasks evaluate models with a fixed number of top-ranked predictions (say F-score @5 and @10), existing keyphrase generation studies have to over-generate phrases by means of beam search (commonly with a large beam size, e.g., 150 and 200 in BIBREF3 , BIBREF0 , respectively), a heuristic search algorithm that returns INLINEFORM0 approximate optimal sequences. For the One2One setting, each returned sequence is a unique phrase itself. But for One2Seq, each produced sequence contains several phrases and additional processes BIBREF2 are needed to obtain the final unique (ordered) phrase list.
It is worth noting that the time complexity of beam search is INLINEFORM0 , where INLINEFORM1 is the beam width, and INLINEFORM2 is the maximum length of generated sequences. Therefore the exhaustive decoding is generally very computationally expensive, especially for One2Seq setting where INLINEFORM3 is much larger than in One2One. It is also wasteful as we observe that less than 5% of phrases generated by One2Seq models are unique.
An innate characteristic of keyphrase tasks is that the number of keyphrases varies depending on the document and dataset genre, therefore dynamically outputting a variable number of phrases is a desirable property for keyphrase generation models. Since our proposed model is trained to generate a variable number of phrases as a single sequence joined by delimiters, we can obtain multiple phrases by simply decoding a single sequence for each given source text. The resulting model thus implicitly performs the additional task of dynamically estimating the proper size of the target phrase set: once the model believes that an adequate number of phrases have been generated, it outputs a special token INLINEFORM0 EOS INLINEFORM1 to terminate the decoding process.
One notable attribute of the self-terminating decoding strategy is that, by generating a set of phrases in a single sequence, the model conditions its current generation on all previously generated phrases. Compared to the exhaustive strategy (i.e., phrases being generated independently by beam search in parallel), our model can model the dependency among its output in a more explicit fashion. Additionally, since multiple phrases are decoded as a single sequence, decoding can be performed more efficiently than exhaustive decoding by conducting greedy search or beam search on only the top-scored sequence.

Experiments on Scientific Publications
Our first dataset consists of a collection of scientific publication datasets, namely KP20k, Inspec, Krapivin, NUS, and SemEval, that have been widely used in existing literature BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 . KP20k, for example, was introduced by BIBREF0 and comprises more than half a million scientific publications. For each article, the abstract and title are used as the source text while the author keywords are used as target. The other four datasets contain much fewer articles, and thus used to test transferability of our model (without fine-tuning).
We report our model's performance on the present-keyphrase portion of the KP20k dataset in Table TABREF35 . To compare with previous works, we provide compute INLINEFORM0 and INLINEFORM1 scores. The new proposed F INLINEFORM2 @ INLINEFORM3 metric indicates consistent ranking with INLINEFORM4 for most cases. Due to its target number sensitivity, we find that its value is closer to INLINEFORM5 for KP20k and Krapivin where average target keyphrases is less and closer to INLINEFORM6 for the other three datasets.
From the result we can see that the neural-based models outperform non-neural models by large margins. Our implemented CopyRNN achieves better or comparable performance against the original model, and on NUS and SemEval the advantage is more salient.
As for the proposed models, both INLINEFORM0 and INLINEFORM1 yield comparable results to CopyRNN, indicating that One2Seq paradigm can work well as an alternative option for the keyphrase generation. INLINEFORM2 outperforms INLINEFORM3 on all metrics, suggesting the semantic coverage and orthogonal regularization help the model to generate higher quality keyphrases and achieve better generalizability. To our surprise, on the metric F INLINEFORM4 @10 for KP20k and Krapivin (average number of keyphrases is only 5), where high-recall models like CopyRNN are more favored, INLINEFORM5 is still able to outperform One2One baselines, indicating that the proposed mechanisms for diverse generation are effective.

Generating Variable Number Keyphrases
One key advantage of our proposed model is the capability of predicting the number of keyphrases conditioned on the given source text. We thus conduct a set of experiments on KP20k and StackEx present keyphrase generation tasks, as shown in Table TABREF39 , to study such behavior. We adopt the self-terminating decoding strategy (Section SECREF28 ), and use both F INLINEFORM0 @ INLINEFORM1 and F INLINEFORM2 @ INLINEFORM3 (Section SECREF4 ) to evaluate.
In these experiments, we use beam search as in most Natural Language Generation (NLG) tasks, i.e., only use the top ranked prediction sequence as output. We compare the results with greedy search. Since no existing model is capable of generating variable number of keyphrases, in this subsection we only report performance on such setting from INLINEFORM0 and INLINEFORM1 .
From Table TABREF39 we observe that in the variable number generation setting, greedy search outperforms beam search consistently. This may because beam search tends to generate short and similar sequences. We can also see the resulting F INLINEFORM0 @ INLINEFORM1 scores are generally lower than results reported in previous subsections, this suggests an over-generation decoding strategy may still benefit from achieving higher recall.

Ablation Study
We conduct an ablation experiment to study the effects of orthogonal regularization and semantic coverage mechanism on INLINEFORM0 . As shown in Table TABREF44 , semantic coverage provides significant boost to INLINEFORM1 's performance on all datasets. Orthogonal regularization hurts performance when is solely applied to INLINEFORM2 model. Interestingly, when both components are enabled ( INLINEFORM3 ), the model outperforms INLINEFORM4 by a noticeable margin on all datasets, this suggests the two components help keyphrase generation in a synergetic way. One future direction is to apply orthogonal regularization directly on target encoder, since the regularizer can potentially diversify target representations at phrase level, which may further encourage diverse keyphrase generation in decoder.

Visualizing Diversified Generation
To verify our assumption that target encoding and orthogonal regularization help to boost the diversity of generated sequences, we use two metrics, one quantitative and one qualitative, to measure diversity of generation.
First, we simply calculate the average unique predictions produced by both INLINEFORM0 and INLINEFORM1 in experiments shown in Section SECREF36 . The resulting numbers are 20.38 and 89.70 for INLINEFORM2 and INLINEFORM3 respectively. Second, from the model running on the KP20k validation set, we randomly sample 2000 decoder hidden states at INLINEFORM4 steps following a delimiter ( INLINEFORM5 ) and apply an unsupervised clustering method (t-SNE BIBREF35 ) on them. From the Figure FIGREF46 we can see that hidden states sampled from INLINEFORM6 are easier to cluster while hidden states sampled from INLINEFORM7 yield one mass of vectors with no obvious distinct clusters. Results on both metrics suggest target encoding and orthogonal regularization indeed help diversifying generation of our model.

Experiment Results on KP20k Absent Subset
Generating absent keyphrases on scientific publication datasets is a rather challenging problem. Existing studies often achieve seemingly good performance by measuring recall on tens and sometimes hundreds of keyphrases produced by exhaustive decoding with a large beam size — thus completely ignoring precision.
We report the models' R@10/50 scores on the absent portion of five scientific paper datasets in Table TABREF48 to be in line with previous studies.
The absent keyphrase prediction highly prefers recall-oriented models, therefore CopyRNN with beam size of 200 is innately proper for this task setting. Howerer, from the results we observe that with the help of exhaustive decoding and diverse mechanisms, INLINEFORM0 is able to perform comparably to CopyRNN model, and it generally works better for top predictions. Even though the trend of models' performance somewhat matches what we observe on the present data, we argue that it is hard to compare different models' performance on such scale. We argue that StackEx is better testbeds for absent keyphrase generation.

Figure 1: Overall structure of our proposed model. A represents last states of the bi-directional source encoder; B represents decoder states where target tokens are delimiters; C indicates target encoder states where input tokens are delimiters. During orthogonal regularization, all B states are used; during target encoder training, we maximize the mutual information between states A with each of the C states; red dash arrow indicates a detached path, i.e., do not back propagate through this path.

Table 1: Statistics of datasets we use in this work. Avg# and Var# indicate the mean and variance of numbers of target phrases per data point, respectively.

Table 4: Model performance on STACKEXCHANGE dataset.

Table 5: Model performance on TEXTWORLD ACG dataset.

Table 7: Average number of generated keyphrase on KP20K and TEXTWORLD ACG.

Table 8: Example observations and admissible commands in TEXTWORLD ACG dataset.
----------------------------------------

Question: What were the baselines?

To answer the question, format your response as a `json` object with two fields:

"answer": A string providing a succinct answer to the question, in your own words.
"evidence": An array of strings. Each entry should be a full paragraph from the paper. Together, the evidence should serve as a justification for the answer.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": "BERT and RoBERTa."
  "evidence": ["We compare our approach against two baselines. In Table 1, we compare against BERT. In Table 2, we compare against RoBERTa. Our findings indicate that our approach improves over both baeslines..."]
}

The "answer" field should be roughly 92 characters in length.

Do not include any text in your response other than the json. If the question is unanswerable given the provided excerpts, respond with the single word "null".

To repeat, the question is: What were the baselines?

Answer JSON object:
{"answer": "CopyRNN (Meng et al., 2017), Multi-Task (Ye and Wang, 2018), and TG-Net (Chen et al., 2018b)", "evidence": ["We report our model\u0027s performance on the present-keyphrase portion of the KP20k dataset in Table TABREF35 . To compare with previous works, we provide compute INLINEFORM0 and INLINEFORM1 scores. The new proposed F INLINEFORM2 @ INLINEFORM3 metric indicates consistent ranking with INLINEFORM4 for most cases. Due to its target number sensitivity, we find that its value is closer to INLINEFORM5 for KP20k and Krapivin where average target keyphrases is less and closer to INLINEFORM6 for the other three datasets."]}