You will be shown sections from a scientific research paper, together with a question about the paper. Paragraphs in the paper are separated by newlines. Your task is to answer the question based on the contents of the paper.

Paper:
----------------------------------------
Ranking Sentences for Extractive Summarization with Reinforcement Learning

Single document summarization is the task of producing a shorter version of a document while preserving its principal information content. In this paper we conceptualize extractive summarization as a sentence ranking task and propose a novel training algorithm which globally optimizes the ROUGE evaluation metric through a reinforcement learning objective. We use our algorithm to train a neural summarization model on the CNN and DailyMail datasets and demonstrate experimentally that it outperforms state-of-the-art extractive and abstractive systems when evaluated automatically and by humans.

Introduction
Automatic summarization has enjoyed wide popularity in natural language processing due to its potential for various information access applications. Examples include tools which aid users navigate and digest web content (e.g., news, social media, product reviews), question answering, and personalized recommendation engines. Single document summarization — the task of producing a shorter version of a document while preserving its information content — is perhaps the most basic of summarization tasks that have been identified over the years (see BIBREF0 , BIBREF0 for a comprehensive overview).
Modern approaches to single document summarization are data-driven, taking advantage of the success of neural network architectures and their ability to learn continuous features without recourse to preprocessing tools or linguistic annotations. Abstractive summarization involves various text rewriting operations (e.g., substitution, deletion, reordering) and has been recently framed as a sequence-to-sequence problem BIBREF1 . Central in most approaches BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 is an encoder-decoder architecture modeled by recurrent neural networks. The encoder reads the source sequence into a list of continuous-space representations from which the decoder generates the target sequence. An attention mechanism BIBREF8 is often used to locate the region of focus during decoding.
Extractive systems create a summary by identifying (and subsequently concatenating) the most important sentences in a document. A few recent approaches BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 conceptualize extractive summarization as a sequence labeling task in which each label specifies whether each document sentence should be included in the summary. Existing models rely on recurrent neural networks to derive a meaning representation of the document which is then used to label each sentence, taking the previously labeled sentences into account. These models are typically trained using cross-entropy loss in order to maximize the likelihood of the ground-truth labels and do not necessarily learn to rank sentences based on their importance due to the absence of a ranking-based objective. Another discrepancy comes from the mismatch between the learning objective and the evaluation criterion, namely ROUGE BIBREF13 , which takes the entire summary into account.
In this paper we argue that cross-entropy training is not optimal for extractive summarization. Models trained this way are prone to generating verbose summaries with unnecessarily long sentences and redundant information. We propose to overcome these difficulties by globally optimizing the ROUGE evaluation metric and learning to rank sentences for summary generation through a reinforcement learning objective. Similar to previous work BIBREF9 , BIBREF11 , BIBREF10 , our neural summarization model consists of a hierarchical document encoder and a hierarchical sentence extractor. During training, it combines the maximum-likelihood cross-entropy loss with rewards from policy gradient reinforcement learning to directly optimize the evaluation metric relevant for the summarization task. We show that this global optimization framework renders extractive models better at discriminating among sentences for the final summary; a sentence is ranked high for selection if it often occurs in high scoring summaries.
We report results on the CNN and DailyMail news highlights datasets BIBREF14 which have been recently used as testbeds for the evaluation of neural summarization systems. Experimental results show that when evaluated automatically (in terms of ROUGE), our model outperforms state-of-the-art extractive and abstractive systems. We also conduct two human evaluations in order to assess (a) which type of summary participants prefer (we compare extractive and abstractive systems) and (b) how much key information from the document is preserved in the summary (we ask participants to answer questions pertaining to the content in the document by reading system summaries). Both evaluations overwhelmingly show that human subjects find our summaries more informative and complete.
Our contributions in this work are three-fold: a novel application of reinforcement learning to sentence ranking for extractive summarization; corroborated by analysis and empirical results showing that cross-entropy training is not well-suited to the summarization task; and large scale user studies following two evaluation paradigms which demonstrate that state-of-the-art abstractive systems lag behind extractive ones when the latter are globally trained.

Summarization as Sentence Ranking
Given a document D consisting of a sequence of sentences INLINEFORM0 , an extractive summarizer aims to produce a summary INLINEFORM1 by selecting INLINEFORM2 sentences from D (where INLINEFORM3 ). For each sentence INLINEFORM4 , we predict a label INLINEFORM5 (where 1 means that INLINEFORM6 should be included in the summary) and assign a score INLINEFORM7 quantifying INLINEFORM8 's relevance to the summary. The model learns to assign INLINEFORM9 when sentence INLINEFORM10 is more relevant than INLINEFORM11 . Model parameters are denoted by INLINEFORM12 . We estimate INLINEFORM13 using a neural network model and assemble a summary INLINEFORM14 by selecting INLINEFORM15 sentences with top INLINEFORM16 scores.
Our architecture resembles those previously proposed in the literature BIBREF9 , BIBREF10 , BIBREF11 . The main components include a sentence encoder, a document encoder, and a sentence extractor (see the left block of Figure FIGREF2 ) which we describe in more detail below.

Sentence Ranking with Reinforcement Learning
Reinforcement learning BIBREF28 has been proposed as a way of training sequence-to-sequence generation models in order to directly optimize the metric used at test time, e.g., BLEU or ROUGE BIBREF29 . We adapt reinforcement learning to our formulation of extractive summarization to rank sentences for summary generation. We propose an objective function that combines the maximum-likelihood cross-entropy loss with rewards from policy gradient reinforcement learning to globally optimize ROUGE. Our training algorithm allows to explore the space of possible summaries, making our model more robust to unseen data. As a result, reinforcement learning helps extractive summarization in two ways: (a) it directly optimizes the evaluation metric instead of maximizing the likelihood of the ground-truth labels and (b) it makes our model better at discriminating among sentences; a sentence is ranked high for selection if it often occurs in high scoring summaries.

Training with High Probability Samples
Computing the expectation term in Equation ( EQREF10 ) is prohibitive, since there is a large number of possible extracts. In practice, we approximate the expected gradient using a single sample INLINEFORM0 from INLINEFORM1 for each training example in a batch: DISPLAYFORM0 
Presented in its original form, the REINFORCE algorithm starts learning with a random policy which can make model training challenging for complex tasks like ours where a single document can give rise to a very large number of candidate summaries. We therefore limit the search space of INLINEFORM0 in Equation ( EQREF12 ) to the set of largest probability samples INLINEFORM1 . We approximate INLINEFORM2 by the INLINEFORM3 extracts which receive highest ROUGE scores. More concretely, we assemble candidate summaries efficiently by first selecting INLINEFORM4 sentences from the document which on their own have high ROUGE scores. We then generate all possible combinations of INLINEFORM5 sentences subject to maximum length INLINEFORM6 and evaluate them against the gold summary. Summaries are ranked according to F INLINEFORM7 by taking the mean of INLINEFORM8 , INLINEFORM9 , and INLINEFORM10 . INLINEFORM11 contains these top INLINEFORM12 candidate summaries. During training, we sample INLINEFORM13 from INLINEFORM14 instead of INLINEFORM15 .
ranzato-arxiv15-bias proposed an alternative to REINFORCE called MIXER (Mixed Incremental Cross-Entropy Reinforce) which first pretrains the model with the cross-entropy loss using ground truth labels and then follows a curriculum learning strategy BIBREF30 to gradually teach the model to produce stable predictions on its own. In our experiments MIXER performed worse than the model of nallapati17 just trained on collective labels. We conjecture that this is due to the unbounded nature of our ranking problem. Recall that our model assigns relevance scores to sentences rather than words. The space of sentential representations is vast and fairly unconstrained compared to other prediction tasks operating with fixed vocabularies BIBREF31 , BIBREF7 , BIBREF32 . Moreover, our approximation of the gradient allows the model to converge much faster to an optimal policy. Advantageously, we do not require an online reward estimator, we pre-compute INLINEFORM0 , which leads to a significant speedup during training compared to MIXER BIBREF29 and related training schemes BIBREF33 .

Experimental Setup
In this section we present our experimental setup for assessing the performance of our model which we call Refresh as a shorthand for REinFoRcement Learning-based Extractive Summarization. We describe our datasets, discuss implementation details, our evaluation protocol, and the systems used for comparison.

Related Work
Traditional summarization methods manually define features to rank sentences for their salience in order to identify the most important sentences in a document or set of documents BIBREF42 , BIBREF43 , BIBREF44 , BIBREF45 , BIBREF46 , BIBREF47 . A vast majority of these methods learn to score each sentence independently BIBREF48 , BIBREF49 , BIBREF50 , BIBREF51 , BIBREF52 , BIBREF53 , BIBREF54 and a summary is generated by selecting top-scored sentences in a way that is not incorporated into the learning process. Summary quality can be improved heuristically, BIBREF55 , via max-margin methods BIBREF56 , BIBREF57 , or integer-linear programming BIBREF58 , BIBREF59 , BIBREF60 , BIBREF61 , BIBREF62 .
Recent deep learning methods BIBREF63 , BIBREF64 , BIBREF9 , BIBREF10 learn continuous features without any linguistic preprocessing (e.g., named entities). Like traditional methods, these approaches also suffer from the mismatch between the learning objective and the evaluation criterion (e.g., ROUGE) used at the test time. In comparison, our neural model globally optimizes the ROUGE evaluation metric through a reinforcement learning objective: sentences are highly ranked if they occur in highly scoring summaries.
Reinforcement learning has been previously used in the context of traditional multi-document summarization as a means of selecting a sentence or a subset of sentences from a document cluster. Ryang:2012 cast the sentence selection task as a search problem. Their agent observes a state (e.g., a candidate summary), executes an action (a transition operation that produces a new state selecting a not-yet-selected sentence), and then receives a delayed reward based on INLINEFORM0 . Follow-on work BIBREF65 extends this approach by employing ROUGE as part of the reward function, while hens-gscl15 further experiment with INLINEFORM1 -learning. MollaAliod:2017:ALTA2017 has adapt this approach to query-focused summarization. Our model differs from these approaches both in application and formulation. We focus solely on extractive summarization, in our case states are documents (not summaries) and actions are relevance scores which lead to sentence ranking (not sentence-to-sentence transitions). Rather than employing reinforcement learning for sentence selection, our algorithm performs sentence ranking using ROUGE as the reward function.
The REINFORCE algorithm BIBREF15 has been shown to improve encoder-decoder text-rewriting systems by allowing to directly optimize a non-differentiable objective BIBREF29 , BIBREF31 , BIBREF7 or to inject task-specific constraints BIBREF32 , BIBREF66 . However, we are not aware of any attempts to use reinforcement learning for training a sentence ranker in the context of extractive summarization.

Figure 1: Extractive summarization model with reinforcement learning: a hierarchical encoder-decoder model ranks sentences for their extract-worthiness and a candidate summary is assembled from the top ranked sentences; the REWARD generator compares the candidate against the gold summary to give a reward which is used in the REINFORCE algorithm (Williams, 1992) to update the model.

Table 1: An abridged CNN article (only first 15 out of 31 sentences are shown) and its “story highlights”. The latter are typically written by journalists to allow readers to quickly gather information on stories. Highlights are often used as gold standard abstractive summaries in the summarization literature.

Figure 2: Summaries produced by the LEAD baseline, the abstractive system of See et al. (2017) and REFRESH for a CNN (test) article. GOLD presents the human-authored summary; the bottom block shows manually written questions using the gold summary and their answers in parentheses.

Table 2: Results on the CNN and DailyMail test sets. We report ROUGE-1 (R1), ROUGE-2 (R2), and ROUGE-L (RL) F1 scores. Extractive systems are in the first block and abstractive systems in the second. Table cells are filled with — whenever results are not available. Models marked with ∗ are not directly comparable to ours as they are based on an anonymized version of the dataset.

Table 3: System ranking and QA-based evaluations. Rankings (1st, 2nd, 3rd and 4th) are shown as proportions. Rank 1 is the best and Rank 4, the worst. The column QA shows the percentage of questions that participants answered correctly by reading system summaries.
----------------------------------------

Question: What are the baselines?

To answer the question, format your response as a `json` object with two fields:

"answer": A string providing a succinct answer to the question, in your own words.
"evidence": An array of strings. Each entry should be a full paragraph from the paper. Together, the evidence should serve as a justification for the answer.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": "BERT and RoBERTa."
  "evidence": ["We compare our approach against two baselines. In Table 1, we compare against BERT. In Table 2, we compare against RoBERTa. Our findings indicate that our approach improves over both baeslines..."]
}

The "answer" field should be roughly 320 characters in length.

Do not include any text in your response other than the json. If the question is unanswerable given the provided excerpts, respond with the single word "null".

To repeat, the question is: What are the baselines?

Answer JSON object:
{"answer": "Answer with content missing: (Experimental Setup missing subsections)\nTo be selected: We compared REFRESH against a baseline which simply selects the first m leading sentences from each document (LEAD) and two neural models similar to ours (see left block in Figure 1), both trained with cross-entropy loss.\nAnswer: LEAD", "evidence": ["Experimental Setup", "In this section we present our experimental setup for assessing the performance of our model which we call Refresh as a shorthand for REinFoRcement Learning-based Extractive Summarization. We describe our datasets, discuss implementation details, our evaluation protocol, and the systems used for comparison."]}