You will be shown sections from a scientific research paper, together with a question about the paper. Your task is to answer the question based on the contents of the paper. Your response should be a `json` object with two fields:

"answer": An array of strings extracted directly from the paper which, collectively, answer the question.
"evidence": An array of strings. Each should be an excerpt from the paper, in which one or more of the extracted answers can be found.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": ["BERT", "RoBERTa"],
  "evidence": ["In our experiments, we compare the performance of our model against BERT and RoBERTa."]
}

Do not include any text in your response other than the json.

If the question is unanswerable given the provided excerpts, respond with the single word "null".

Paper: STEP: Sequence-to-Sequence Transformer Pre-training for Document Summarization

Abstractive summarization aims to rewrite a long document to its shorter form, which is usually modeled as a sequence-to-sequence (Seq2Seq) learning problem. Seq2Seq Transformers are powerful models for this problem. Unfortunately, training large Seq2Seq Transformers on limited supervised summarization data is challenging. We, therefore, propose STEP (as shorthand for Sequence-to-Sequence Transformer Pre-training), which can be trained on large scale unlabeled documents. Specifically, STEP is pre-trained using three different tasks, namely sentence reordering, next sentence generation, and masked document generation. Experiments on two summarization datasets show that all three tasks can improve performance upon a heavily tuned large Seq2Seq Transformer which already includes a strong pre-trained encoder by a large margin. By using our best task to pre-train STEP, we outperform the best published abstractive model on CNN/DailyMail by 0.8 ROUGE-2 and New York Times by 2.4 ROUGE-2.

Related Work ::: Abstractive Summarization
The goal of abstractive summarization is to generate summaries by rewriting a document, which is a sequence-to-sequence learning problem. seq2seq attentive LSTMs BIBREF8, BIBREF24 are employed in nallapati2016abstractive. Even these models are extended with copy mechanism BIBREF25, coverage model BIBREF11 and reinforcement learning BIBREF26, their results are still very close to that of Lead3 which selects the leading three sentences of a document as its summary. One possible reason is that LSTMs without pre-training are not powerful enough. liu2019text used a seq2seq Transformer model with its encoder initialized with a pre-trained Transformer (i.e., BERT; BIBREF2) and achieved the state-of-the-art performance. Our work goes one step further, we propose a method to pre-train the decoder together with the encoder and then initialize both the encoder and decoder of a summarization model with the pre-trained Transformers.
There is also a line of work that bridges extractive and abstractive models with reinforcement learning BIBREF27, attention fusion BIBREF28 and bottom-up attention BIBREF29, while our model is conceptually simpler.

Sequence-to-Sequence Transformer Pre-training ::: Pre-training Tasks
Training a seq2seq Transformer model on a summarization dataset from scratch is difficult due to the limited number of document-summary pairs. Pre-trained Transformer encoders such as BERT BIBREF2 and RoBERTa BIBREF4 have achieved great success in many natural language understanding tasks. Therefore, we first initialize the encoder of our seq2seq Transformer summarization model Step with an existing pre-trained Transformer encoder (i.e., RoBERTa) to enhance its language understanding capabilities. To help Step gain language generation capabilities and the abilities of associating generated text with encoder outputs, we continue to pre-train it on unlabeled text. In the following, we describe our pre-training tasks.

Sequence-to-Sequence Transformer Pre-training ::: Pre-training Tasks ::: Masked Document Generation
The third task we consider is Masked Document Generation (MDG) that learns to recover a document with a masked span of tokens (see Figure FIGREF5). For simplicity, a document consisting of a sequence of tokens is denoted as $X=(x_1, x_2, \cdots , x_{|X|})$. We randomly sample the length of the span $l$ from a discrete uniform distribution $\mathcal {U}(a, b)$ and the span start position $k$ from another discrete uniform distribution $\mathcal {U}(1, |X|-l+1)$ (see Section SECREF4 for more details). Thus, $\mathcal {M}=(x_k, x_{k+1}, \cdots , x_{k+l-1})$ is the text span to be masked.
One straightforward masking strategy is to replace each token residing in $\mathcal {M}$ with a special [MASK] token. However, we refrain from doing so because of the following three reasons. Usually, [MASK] tokens will not appear in downstream tasks. Second, entirely masking a continuous sub-sequence of $X$ may make the whole document incomprehensible, which might be too challenging for our model to learn. Third, similar to SR, avoiding replacing every token with [MASK] also helps our model learn the ability of copying tokens from the input while preserving the ability of generating novel tokens.
In the sub-sequence $\mathcal {M}$, each token is processed with one of the three strategies: 1) replaced with the [MASK] token; 2) replaced with a random token; 3) remain unchanged. Inspired by BERT BIBREF2, for 80% tokens, we follow strategy 1). In 10% of cases, we employ strategy 2) and we use strategy 3) for the remaining 10% of cases. Let $\hat{X}_M$ denote the document after the application of our masking strategy. We could create infinite amount of $\langle \hat{X}_M,X\rangle $ pairs to train Step.
During pre-training, we could also employ all the three tasks (i.e., SR, NSG, MDG) together. For each training batch, we randomly choose one task and each task is used for $1/3$ of the time.

Experimental Setup
In this section, we present the experimental setup for evaluating our summarization models. We first introduce the datasets used for our experiments. Then we describe training details of our models as well as our evaluation protocols.

Experimental Setup ::: Datasets ::: CNNDM
CNNDM contains news articles and their associated highlights (i.e., summaries) collected from the CNN and Daily Mail Online websites. Following previous work BIBREF11, BIBREF22, BIBREF23, we use the non-anonymized version of CNNDM. Specifically, we preprocess the dataset with the publicly available scripts provided by see2017get and obtain 287,226 document-summary pairs for training, 13,368 for validation and 11,490 for test.

Experimental Setup ::: Datasets ::: NYT
The NYT dataset is a collection of articles along with multi-sentence summaries written by library scientists. We closely follow the preprocessing procedures described in durrett2016learning and liu2019text. The test set is constructed by including all articles published on January 1, 2017 or later, which contains 9,076 articles. The remaining 100,834 articles are split into a training set of 96,834 examples and a validation set of 4,000 examples. As in BIBREF36, we also remove articles whose summaries contain less than 50 words from the test set, and the resulting test set contains 3,452 examples.

Experimental Setup ::: Datasets ::: GIGA-CM
To pre-train our model with the tasks introduced in Section SECREF9, following the procedures in BIBREF22, we created the GIGA-CM dataset, which contains only unlabeled documents. The training set of GIGA-CM is composed of 6,521,658 documents sampled from the English Gigaword dataset and the training documents in CNNDM. We used the 13,368 documents in the validation split of CNNDM as its validation set. Note that the Gigaword dataset overlaps with the NYT dataset and we therefore exclude the test set of NYT from the training set of GIGA-CM.
For CNNDM, NYT and GIGA-CM datasets, we segment and tokenize documents and/or summaries (GIGA-CM only contains documents) using the Stanford CoreNLP toolkit BIBREF37. To reduce the vocabulary size, we further apply the UTF8 based BPE BIBREF38 introduced in GPT-2 BIBREF39 to all datasets. As a common wisdom in abstractive summarization, documents and summaries in CNNDM and NYT are usually truncated to 512 and 256 tokens, respectively.
We leverage unlabeled documents differently for different pre-training tasks (see Section SECREF9). We first split each document into 512 token segments if it contains more than 512 tokens (segments or documents with less than 512 tokens are removed). In Sentence Reordering (SR) and Masked Document Generation (MDG), we use the segment after transformation to predict the original segment. We set the minimum masked length $a=100$ and the maximum masked length $b=256$ in MDG. In Next Sentence Generation (NSG), each segment is used to predict its next 256 tokens.

Results ::: Automatic Evaluation
The results on the CNNDM are summarized in Table TABREF25. The first and second blocks show results of previous extractive and abstractive models, respectively. Results of Step are all listed in the third block. Lead3 is a baseline which simply takes the first three sentences of a document as its summary. BERTExt BIBREF23 is an extractive model fine-tuning on BERT BIBREF2 that outperforms other extractive systems. PTGen BIBREF11, DRM BIBREF26, and DCA BIBREF42 are sequence-to-sequence learning based models extended with copy and coverage mechanism, reinforcement learning, and deep communicating agents individually. BottomUp BIBREF29 assisted summary generation with a word prediction model. BERTAbs BIBREF23 and UniLM BIBREF43 are both pre-training based seq2seq summarization models. We also implemented three abstractive models as our baselines. Transformer-S2S is 6-layer seq2seq Transformer BIBREF9 with random initialization. When we replaced the encoder of Transformer-S2S with $\text{RoBERTa}_\text{BASE}$ BIBREF4, $\text{RoBERTa}_\text{BASE}$-S2S outperforms Transformer-S2S by nearly 2 ROUGE, which demonstrates the effectiveness of pre-trained models. With even larger pre-trained model $\text{RoBERTa}_\text{LARGE}$, $\text{RoBERTa}$-S2S is comparable with the best published abstractive model UniLM BIBREF43.
Based on $\text{RoBERTa}$-S2S (the sizes of Step and $\text{RoBERTa}$-S2S are identical), we study the effect of different pre-training tasks (see Section SECREF9). We first pre-train Step on unlabeled documents of CNNDM training split to get quick feedback, denoted as Step (in-domain). From the top part of the third block in Table TABREF25, we can see that Sentence Reordering (SR), Next Sentence Generation (NSG) and Masked Document Generation (MDG) can all improve $\text{RoBERTa}$-S2S significantly measured by the ROUGE script. Note that according to the ROUGE script, $\pm 0.22$ ROUGE almost always means a significant difference with $p < 0.05$. Interesting, even Step is pre-trained on 230 million words, it outperforms UniLM that is pre-trained on 3,000 million words BIBREF43. When we pre-train Step on even larger dataset (i.e., GIGA-CM), the results are further improved and Step outperforms all models in comparison, as listed in the bottom part of Table TABREF25.
Table TABREF26 presents results on NYT dataset. Following the same evaluation protocol as BIBREF36, we adopted the limited-length recall based ROUGE, where we truncated the predicted summaries to the length of the gold ones. Again, the first and second blocks show results of previous extractive and abstractive models, respectively. Results of Step are listed in the third block. Similar to the trends in CNNDM, Step leads significant performance gains (with $p<0.05$) compared to all other models in Table TABREF26.
Among all three pre-training tasks, SR works slightly better than the other two tasks (i.e., NSG and MDG). We also tried to randomly use all the three tasks during training with 1/3 probability each (indicated as ALL). Interesting, we observed that, in general, All outperforms all three tasks when employing unlabeled documents of training splits of CNNDM or NYT, which might be due to limited number of unlabeled documents of the training splits. After adding more data (i.e., GIAG-CM) to pre-training, SR consistently achieves highest ROUGE-2 on both CNNDM and NYT. We conclude that SR is the most effective task for pre-training since sentence reordering task requires comprehensively understanding a document in a wide coverage, going beyond individual words and sentences, which is highly close to the essense of abstractive document summarization.

Conclusion
We proposed Step, a seq2seq transformer pre-training approach, for abstractive document summarization. Specifically, three pre-training tasks are designed, sentence reordering, next sentence generation, and masked document generation. When we only employ the unlabeled documents in the training splits of summarization datasets to pre-training Step with our proposed tasks, the summarization model based on the pre-trained Step outperforms the best published abstractive system. Involving large scale data to pre-training leads to larger performance gains. By using the best performing pre-training task, Step achieves 0.8 absolute ROUGE-2 improvements on CNN/DailyMail and 2.4 absolute ROUGE-2 improvements on New York Times. In the future, we would like to investigate other tasks to pre-train the seq2seq transformer model. Pre-training for unsupervised abstractive summarization is also an interesting direction and worth exploration.

Figure 1: Pre-training of STEP. The document (x1, x2, · · · , x8) contains three sentences (i.e., SENT. 1, SENT. 2 and SENT. 3). STEP adopts the SEQ2SEQ Transformer architecture. It takes the transformed document (i.e., a shuffled document, the first segment of a document, or a masked document) as input and learns to recover the original document (or part of the original document) by generation. SR: Sentence Reordering; NSG: Next Sentence Generation; MDG: Masked Document Generation.

Table 1: Results on the test split of CNNDM using fulllength F1 based ROUGE-1 (R-1), ROUGE-2 (R-2) and ROUGE-L (R-L). ∗ indicates significant improvements (p < 0.05 measured with the ROUGE script) compared to models in the first two blocks.

Table 2: Results on the test set of NYT dataset using limited-length recall based ROUGE. ∗ indicates significant improvements (p < 0.05 measured with the ROUGE script) to models in the first two blocks.

Table 3: Human evaluation results: proportions of system rankings. MR: mean rank (the lower the better).

Question: Which of the three pretraining tasks is the most helpful?
{"answer": ["SR"], "evidence": ["Among all three pre-training tasks, SR works slightly better than the other two tasks (i.e., NSG and MDG). We also tried to randomly use all the three tasks during training with 1/3 probability each (indicated as ALL). Interesting, we observed that, in general, All outperforms all three tasks when employing unlabeled documents of training splits of CNNDM or NYT, which might be due to limited number of unlabeled documents of the training splits. After adding more data (i.e., GIAG-CM) to pre-training, SR consistently achieves highest ROUGE-2 on both CNNDM and NYT. We conclude that SR is the most effective task for pre-training since sentence reordering task requires comprehensively understanding a document in a wide coverage, going beyond individual words and sentences, which is highly close to the essense of abstractive document summarization."]}