You will be shown sections from a scientific research paper, together with a question about the paper. Your task is to answer the question based on the contents of the paper. Your response should be a `json` object with two fields:

"answer": An array of strings extracted directly from the paper which, collectively, answer the question.
"evidence": An array of strings. Each should be an excerpt from the paper, in which one or more of the extracted answers can be found.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": ["BERT", "RoBERTa"],
  "evidence": ["In our experiments, we compare the performance of our model against BERT and RoBERTa."]
}

Do not include any text in your response other than the json.

If the question is unanswerable given the provided excerpts, respond with the single word "null".

Paper: Quasi-Recurrent Neural Networks

Recurrent neural networks are a powerful tool for modeling sequential data, but the dependence of each timestep's computation on the previous timestep's output limits parallelism and makes RNNs unwieldy for very long sequences. We introduce quasi-recurrent neural networks (QRNNs), an approach to neural sequence modeling that alternates convolutional layers, which apply in parallel across timesteps, and a minimalist recurrent pooling function that applies in parallel across channels. Despite lacking trainable recurrent layers, stacked QRNNs have better predictive accuracy than stacked LSTMs of the same hidden size. Due to their increased parallelism, they are up to 16 times faster at train and test time. Experiments on language modeling, sentiment classification, and character-level neural machine translation demonstrate these advantages and underline the viability of QRNNs as a basic building block for a variety of sequence tasks.

Variants
Motivated by several common natural language tasks, and the long history of work on related architectures, we introduce several extensions to the stacked QRNN described above. Notably, many extensions to both recurrent and convolutional models can be applied directly to the QRNN as it combines elements of both model types.
Regularization An important extension to the stacked QRNN is a robust regularization scheme inspired by recent work in regularizing LSTMs.
The need for an effective regularization method for LSTMs, and dropout's relative lack of efficacy when applied to recurrent connections, led to the development of recurrent dropout schemes, including variational inference–based dropout BIBREF13 and zoneout BIBREF14 . These schemes extend dropout to the recurrent setting by taking advantage of the repeating structure of recurrent networks, providing more powerful and less destructive regularization.
Variational inference–based dropout locks the dropout mask used for the recurrent connections across timesteps, so a single RNN pass uses a single stochastic subset of the recurrent weights. Zoneout stochastically chooses a new subset of channels to “zone out” at each timestep; for these channels the network copies states from one timestep to the next without modification.
As QRNNs lack recurrent weights, the variational inference approach does not apply. Thus we extended zoneout to the QRNN architecture by modifying the pooling function to keep the previous pooling state for a stochastic subset of channels. Conveniently, this is equivalent to stochastically setting a subset of the QRNN's INLINEFORM0 gate channels to 1, or applying dropout on INLINEFORM1 : DISPLAYFORM0 
 Thus the pooling function itself need not be modified at all. We note that when using an off-the-shelf dropout layer in this context, it is important to remove automatic rescaling functionality from the implementation if it is present. In many experiments, we also apply ordinary dropout between layers, including between word embeddings and the first QRNN layer.
Densely-Connected Layers We can also extend the QRNN architecture using techniques introduced for convolutional networks. For sequence classification tasks, we found it helpful to use skip-connections between every QRNN layer, a technique termed “dense convolution” by BIBREF15 . Where traditional feed-forward or convolutional networks have connections only between subsequent layers, a “DenseNet” with INLINEFORM0 layers has feed-forward or convolutional connections between every pair of layers, for a total of INLINEFORM1 . This can improve gradient flow and convergence properties, especially in deeper networks, although it requires a parameter count that is quadratic in the number of layers.
When applying this technique to the QRNN, we include connections between the input embeddings and every QRNN layer and between every pair of QRNN layers. This is equivalent to concatenating each QRNN layer's input to its output along the channel dimension before feeding the state into the next layer. The output of the last layer alone is then used as the overall encoding result.
Encoder–Decoder Models To demonstrate the generality of QRNNs, we extend the model architecture to sequence-to-sequence tasks, such as machine translation, by using a QRNN as encoder and a modified QRNN, enhanced with attention, as decoder. The motivation for modifying the decoder is that simply feeding the last encoder hidden state (the output of the encoder's pooling layer) into the decoder's recurrent pooling layer, analogously to conventional recurrent encoder–decoder architectures, would not allow the encoder state to affect the gate or update values that are provided to the decoder's pooling layer. This would substantially limit the representational power of the decoder.
Instead, the output of each decoder QRNN layer's convolution functions is supplemented at every timestep with the final encoder hidden state. This is accomplished by adding the result of the convolution for layer INLINEFORM0 (e.g., INLINEFORM1 , in INLINEFORM2 ) with broadcasting to a linearly projected copy of layer INLINEFORM3 's last encoder state (e.g., INLINEFORM4 , in INLINEFORM5 ): DISPLAYFORM0 
 where the tilde denotes that INLINEFORM0 is an encoder variable. Encoder–decoder models which operate on long sequences are made significantly more powerful with the addition of soft attention BIBREF3 , which removes the need for the entire input representation to fit into a fixed-length encoding vector. In our experiments, we computed an attentional sum of the encoder's last layer's hidden states. We used the dot products of these encoder hidden states with the decoder's last layer's un-gated hidden states, applying a INLINEFORM1 along the encoder timesteps, to weight the encoder states into an attentional sum INLINEFORM2 for each decoder timestep. This context, and the decoder state, are then fed into a linear layer followed by the output gate: DISPLAYFORM0 
 where INLINEFORM0 is the last layer.
While the first step of this attention procedure is quadratic in the sequence length, in practice it takes significantly less computation time than the model's linear and convolutional layers due to the simple and highly parallel dot-product scoring function.

Experiments
We evaluate the performance of the QRNN on three different natural language tasks: document-level sentiment classification, language modeling, and character-based neural machine translation. Our QRNN models outperform LSTM-based models of equal hidden size on all three tasks while dramatically improving computation speed. Experiments were implemented in Chainer BIBREF16 .

Sentiment Classification
We evaluate the QRNN architecture on a popular document-level sentiment classification benchmark, the IMDb movie review dataset BIBREF17 . The dataset consists of a balanced sample of 25,000 positive and 25,000 negative reviews, divided into equal-size train and test sets, with an average document length of 231 words BIBREF18 . We compare only to other results that do not make use of additional unlabeled data (thus excluding e.g., BIBREF19 ).
Our best performance on a held-out development set was achieved using a four-layer densely-connected QRNN with 256 units per layer and word vectors initialized using 300-dimensional cased GloVe embeddings BIBREF20 . Dropout of 0.3 was applied between layers, and we used INLINEFORM0 regularization of INLINEFORM1 . Optimization was performed on minibatches of 24 examples using RMSprop BIBREF21 with learning rate of INLINEFORM2 , INLINEFORM3 , and INLINEFORM4 .
Small batch sizes and long sequence lengths provide an ideal situation for demonstrating the QRNN's performance advantages over traditional recurrent architectures. We observed a speedup of 3.2x on IMDb train time per epoch compared to the optimized LSTM implementation provided in NVIDIA's cuDNN library. For specific batch sizes and sequence lengths, a 16x speed gain is possible. Figure FIGREF15 provides extensive speed comparisons.
In Figure FIGREF12 , we visualize the hidden state vectors INLINEFORM0 of the final QRNN layer on part of an example from the IMDb dataset. Even without any post-processing, changes in the hidden state are visible and interpretable in regards to the input. This is a consequence of the elementwise nature of the recurrent pooling function, which delays direct interaction between different channels of the hidden state until the computation of the next QRNN layer.

Character-level Neural Machine Translation
We evaluate the sequence-to-sequence QRNN architecture described in SECREF5 on a challenging neural machine translation task, IWSLT German–English spoken-domain translation, applying fully character-level segmentation. This dataset consists of 209,772 sentence pairs of parallel training data from transcribed TED and TEDx presentations, with a mean sentence length of 103 characters for German and 93 for English. We remove training sentences with more than 300 characters in English or German, and use a unified vocabulary of 187 Unicode code points.
Our best performance on a development set (TED.tst2013) was achieved using a four-layer encoder–decoder QRNN with 320 units per layer, no dropout or INLINEFORM0 regularization, and gradient rescaling to a maximum magnitude of 5. Inputs were supplied to the encoder reversed, while the encoder convolutions were not masked. The first encoder layer used convolutional filter width INLINEFORM1 , while the other encoder layers used INLINEFORM2 . Optimization was performed for 10 epochs on minibatches of 16 examples using Adam BIBREF28 with INLINEFORM3 , INLINEFORM4 , INLINEFORM5 , and INLINEFORM6 . Decoding was performed using beam search with beam width 8 and length normalization INLINEFORM7 . The modified log-probability ranking criterion is provided in the appendix.
Results using this architecture were compared to an equal-sized four-layer encoder–decoder LSTM with attention, applying dropout of 0.2. We again optimized using Adam; other hyperparameters were equal to their values for the QRNN and the same beam search procedure was applied. Table TABREF17 shows that the QRNN outperformed the character-level LSTM, almost matching the performance of a word-level attentional baseline.

Related Work
Exploring alternatives to traditional RNNs for sequence tasks is a major area of current research. Quasi-recurrent neural networks are related to several such recently described models, especially the strongly-typed recurrent neural networks (T-RNN) introduced by BIBREF12 . While the motivation and constraints described in that work are different, BIBREF12 's concepts of “learnware” and “firmware” parallel our discussion of convolution-like and pooling-like subcomponents. As the use of a fully connected layer for recurrent connections violates the constraint of “strong typing”, all strongly-typed RNN architectures (including the T-RNN, T-GRU, and T-LSTM) are also quasi-recurrent. However, some QRNN models (including those with attention or skip-connections) are not “strongly typed”. In particular, a T-RNN differs from a QRNN as described in this paper with filter size 1 and f-pooling only in the absence of an activation function on INLINEFORM0 . Similarly, T-GRUs and T-LSTMs differ from QRNNs with filter size 2 and fo- or ifo-pooling respectively in that they lack INLINEFORM1 on INLINEFORM2 and use INLINEFORM3 rather than sigmoid on INLINEFORM4 .
The QRNN is also related to work in hybrid convolutional–recurrent models. BIBREF31 apply CNNs at the word level to generate INLINEFORM0 -gram features used by an LSTM for text classification. BIBREF32 also tackle text classification by applying convolutions at the character level, with a stride to reduce sequence length, then feeding these features into a bidirectional LSTM. A similar approach was taken by BIBREF10 for character-level machine translation. Their model's encoder uses a convolutional layer followed by max-pooling to reduce sequence length, a four-layer highway network, and a bidirectional GRU. The parallelism of the convolutional, pooling, and highway layers allows training speed comparable to subword-level models without hard-coded text segmentation.
The QRNN encoder–decoder model shares the favorable parallelism and path-length properties exhibited by the ByteNet BIBREF33 , an architecture for character-level machine translation based on residual convolutions over binary trees. Their model was constructed to achieve three desired properties: parallelism, linear-time computational complexity, and short paths between any pair of words in order to better propagate gradient signals.

Beam search ranking criterion
The modified log-probability ranking criterion we used in beam search for translation experiments is: DISPLAYFORM0 
 where INLINEFORM0 is a length normalization parameter BIBREF34 , INLINEFORM1 is the INLINEFORM2 th output character, and INLINEFORM3 is a “target length” equal to the source sentence length plus five characters. This reduces at INLINEFORM4 to ordinary beam search with probabilities: DISPLAYFORM0 
 and at INLINEFORM0 to beam search with probabilities normalized by length (up to the target length): DISPLAYFORM0 
 Conveniently, this ranking criterion can be computed at intermediate beam-search timesteps, obviating the need to apply a separate reranking on complete hypotheses.

Figure 1: Block diagrams showing the computation structure of the QRNN compared with typical LSTM and CNN architectures. Red signifies convolutions or matrix multiplications; a continuous block means that those computations can proceed in parallel. Blue signifies parameterless functions that operate in parallel along the channel/feature dimension. LSTMs can be factored into (red) linear blocks and (blue) elementwise blocks, but computation at each timestep still depends on the results from the previous timestep.

Figure 2: The QRNN encoder–decoder architecture used for machine translation experiments.

Table 1: Accuracy comparison on the IMDb binary sentiment classification task. All of our models use 256 units per layer; all layers other than the first layer, whose filter width may vary, use filter width k = 2. Train times are reported on a single NVIDIA K40 GPU. We exclude semi-supervised models that conduct additional training on the unlabeled portion of the dataset.

Figure 3: Visualization of the final QRNN layer’s hidden state vectors cLt in the IMDb task, with timesteps along the vertical axis. Colors denote neuron activations. After an initial positive statement “This movie is simply gorgeous” (off graph at timestep 9), timestep 117 triggers a reset of most hidden states due to the phrase “not exactly a bad story” (soon after “main weakness is its story”). Only at timestep 158, after “I recommend this movie to everyone, even if you’ve never played the game”, do the hidden units recover.

Table 2: Single model perplexity on validation and test sets for the Penn Treebank language modeling task. Lower is better. “Medium” refers to a two-layer network with 640 or 650 hidden units per layer. All QRNN models include dropout of 0.5 on embeddings and between layers. MC refers to Monte Carlo dropout averaging at test time.

Question: What sentiment classification dataset is used?
{"answer": ["the IMDb movie review dataset BIBREF17"], "evidence": ["We evaluate the QRNN architecture on a popular document-level sentiment classification benchmark, the IMDb movie review dataset BIBREF17 . The dataset consists of a balanced sample of 25,000 positive and 25,000 negative reviews, divided into equal-size train and test sets, with an average document length of 231 words BIBREF18 . We compare only to other results that do not make use of additional unlabeled data (thus excluding e.g., BIBREF19 )."]}