You will be shown sections from a scientific research paper, together with a question about the paper. Your task is to answer the question based on the contents of the paper. Your response should be a `json` object with two fields:

"answer": An array of strings extracted directly from the paper which, collectively, answer the question.
"evidence": An array of strings. Each should be an excerpt from the paper, in which one or more of the extracted answers can be found.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": ["BERT", "RoBERTa"],
  "evidence": ["In our experiments, we compare the performance of our model against BERT and RoBERTa."]
}

Do not include any text in your response other than the json.

If the question is unanswerable given the provided excerpts, respond with the single word "null".

Paper: Elephant in the Room: An Evaluation Framework for Assessing Adversarial Examples in NLP

An adversarial example is an input transformed by small perturbations that machine learning models consistently misclassify. While there are a number of methods proposed to generate adversarial examples for text data, it is not trivial to assess the quality of these adversarial examples, as minor perturbations (such as changing a word in a sentence) can lead to a significant shift in their meaning, readability and classification label. In this paper, we propose an evaluation framework to assess the quality of adversarial examples based on the aforementioned properties. We experiment with five benchmark attacking methods and an alternative approach based on an auto-encoder, and found that these methods generate adversarial examples with poor readability and content preservation. We also learned that there are multiple factors that can influence the attacking performance, such as the the length of text examples and the input domain.

Introduction
Adversarial examples, a term introduced in BIBREF0, are inputs transformed by small perturbations that machine learning models consistently misclassify. The experiments are conducted in the context of computer vision (CV), and the core idea is encapsulated by an illustrative example: after imperceptible noises are added to a panda image, an image classifier predicts, with high confidence, that it is a gibbon. Interestingly, these adversarial examples can also be used to improve the classifier — either as additional training data BIBREF0 or as a regularisation objective BIBREF1 — thus providing motivation for generating effective adversarial examples.
The germ of this paper comes from our investigation of adversarial attack methods for natural language processing (NLP) tasks, e.g. sentiment classification, which drives us to quantify what is an “effective” or “good” adversarial example. In the context of images, a good adversarial example is typically defined according two criteria:
it has successfully fooled the classifier;
it is visually similar to the original example.
In NLP, defining a good adversarial example is a little more involving, because while criterion (b) can be measured with a comparable text similarity metric (e.g. BLEU or edit distance), an adversarial example should also:
be fluent or natural;
preserve its original label.
These two additional criteria are generally irrelevant for images, as adding minor perturbations to an image is unlikely to: (1) create an uninterpretable image (where else changing one word in a sentence can render a sentence incoherent), or (2) change how we perceive the image, say from seeing a panda to a gibbon (but a sentence's sentiment can be reversed by simply adding a negative adverb such as not). Without considering criterion (d), generating adversarial examples in NLP would be trivial, as the model can learn to simply replace a positive adjective (amazing) with a negative one (awful) to attack a sentiment classifier.
To the best of our knowledge, most studies on adversarial example generation in NLP have largely ignored these additional criteria BIBREF2, BIBREF3, BIBREF4, BIBREF5. We believe the lack of a rigorous evaluation framework partially explains why adversarial training for NLP models has not seen the same extent of improvement compared to CV models. As our experiments reveal, examples generated from most attacking methods are successful in fooling the classifier, but their language is often unnatural and the original label is not properly preserved.
The core contribution of our paper is to introduce a systematic, rigorous evaluation framework to assess the quality of adversarial examples for NLP. We focus on sentiment classification as the target task, as it is a popular application that highlights the importance of criteria discussed above. We test a number of attacking methods and also propose an alternative approach (based on an auto-encoder) for generating adversarial examples. We learn that a number of factors can influence the performance of adversarial attacks, including architecture of the classifier, sentence length and input domain.

Related Work
Most existing adversarial attack methods for text inputs are derived from those for image inputs. These methods can be categorised into three types including gradient-based attacks, optimisation-based attacks and model-based attacks.
Gradient-based attacks are mainly white-box attacks that rely on calculating the gradients of the target classifier with respect to the input representation. This class of attacking methods BIBREF6, BIBREF7, BIBREF6 are mainly derived from the fast gradient sign method (FGSM) BIBREF1, and it has been shown to be effective in attacking CV classifiers. However, these gradient-based methods could not be applied to text directly because perturbed word embeddings do not necessarily map to valid words. Other methods such as DeepFool BIBREF8 that rely on perturbing the word embedding space face similar roadblocks. BIBREF5 propose to use nearest neighbour search to find the closest word to the perturbed embedding.
Both optimisation-based and model-based attacks treat adversarial attack as an optimisation problem where the constraints are to maximise the loss of target classifiers and to minimise the difference between original and adversarial examples. Between these two, the former uses optimisation algorithms directly; while the latter trains a seperate model to generate adversarial examples and therefore involves a training process. Some of the most effective attacks for images are achieved by optimisation-based methods, such as the L-BFGS attack BIBREF1 and the C&W attack BIBREF9 in white-box attacks and the ZOO method BIBREF10 in black-box attacks. For texts, the white-box attack HotFlip BIBREF3 and black-box attack DeepWordBug BIBREF11 and TextBugger BIBREF12 are proposed in this category.
In a similar vein, a few model-based attacks have been proposed for images, e.g. BIBREF13 design a generative adversarial network (GAN) to generate the image perturbation from a noise map. The attacking method and target classifier typically form a single large network and the attacking method is trained using the loss from the target classifier. For this reason, it is not very straightforward to use these model-based techniques for text because there is a discontinuity in the network (since words in the adversarial examples are discrete) and so it is not fully differentiable.

Experiments ::: Datasets
We construct three datasets based on IMDB reviews and Yelp reviews. The IMDB dataset is binarised and split into a training and test set, each with 25K reviews (2K reviews from the training set are reserved for development). We filter out any review that has more than 400 tokens, producing the final dataset (imdb400). For Yelp, we binarise the ratings, and create 2 datasets, where we keep only reviews with $\le $ 50 tokens (yelp50) and $\le $200 tokens (yelp200). We randomly partition both datasets into train/dev/test sets (90/5/5 for yelp50; 99/0.5/0.5 for yelp200). For all datasets, we use spaCy for tokenisation. We train and tune target classifiers (see Section SECREF8) using the training and development sets; and evaluate their performance on the original examples in the test sets as well as the adversarial examples generated by attacking methods for the test sets. Note that AutoEncoder also involves a training process, for which we train and tune AutoEncoder using the training and development sets in yelp50, yelp200 and imdb400. Statistics of the three datasets are presented in Table TABREF22. These datasets present a variation in the text lengths (e.g. the average number of words for yelp50, yelp200 and imdb400 is 34, 82 and 195 words respectively), training data size (e.g. the number of training examples for target classifiers for imdb400, yelp50 and yelp200 are 18K, 407K and 2M, respectively) and input domain (e.g. restaurant vs. movie reviews).

Experiments ::: Implementation Details
We use the pre-trained glove.840B.300d embeddings BIBREF20 for all 6 attacking methods. For FGM, FGVM and DeepFool, we tune $\epsilon $, the overshoot hyper-parameter (Section SECREF9) and keep the iterative step $n$ static (5). For TYC, besides $\epsilon $ we also tune the upper limit of flipped words, ranging from 10%–100% of the maximum length. For HotFlip, we tune only the upper limit of flipped words, in the range of $[1, 7]$.
We pre-train AutoEncoder to reconstruct sentences in different datasets as we found that this improves the quality of the generated adversarial examples. During pre-training, we tune batch size, number of layers and number of units, and stop the training after the performance on the development sets stops improving for 20K steps. The model is then initialised with the pre-trained weights and trained based on objectives defined in Section SECREF11. During the training process we tune $\lambda _{ae}$ and $\lambda _{seq}$ while keeping the batch size (32) and learning rate ($1e^{-4}$) fixed. As part of our preliminary study, we also tested different values for the Gumbel-softmax temperature $\tau $ and find that $\tau =0.1$ performs the best. Embeddings are fixed throughout all training processes.
For target classifiers, we tune batch size, learning rate, number of layers, number of units, attention size (BiLSTM$+$A), filter sizes and dropout probability (CNN). For BERT, we use the default fine-tuning hyper-parameter values except for batch size, where we adjust based on memory consumption. Note that after the target classifiers are trained their weights are not updated when training or testing the attacking methods.

Evaluation
We propose both automatic metrics and manual evaluation strategies to assess the quality of adversarial examples, based on four criteria defined in Section SECREF1: (a) attacking performance (i.e. how well they fool the classifier); (b) textual similarity between the original input and the adversarial input; (c) fluency of the adversarial example; and (d) label preservation. Note that the automatic metrics only address the first 3 criteria (a, b and c); we contend that criterion (d) requires manual evaluation, as the judgement of whether the original label is preserved is inherently a human decision.

Evaluation ::: Human Evaluation: Design
Automatic metrics provide a proxy to quantify the quality of the adversarial examples. To validate that these metrics work, we conduct a crowdsourcing experiment on Figure-Eight. Recall that the automatic metrics do not assess sentiment preservation (criterion (d)); we evaluate that aspect here.
We experiment with the 3 best methods (TYC, AutoEncoder and HotFlip) on 2 accuracy thresholds (T0 and T2), using BiLSTM$+$A as the classifier. For each method and threshold, we randomly sample 25 positive-to-negative and 25 negative-to-positive examples. To control for quality, we reserve and annotate 10% of the samples ourselves as control questions. Workers are first presented with 10 control questions as a quiz, and only those who pass the quiz with at least 80% accuracy can continue to work on the task. We display 10 questions per page, where one control question is embedded to continue monitor worker performance. The task is designed such that each control question can only be seen once per worker. We restrict our jobs to workers in United States, United Kingdoms, Australia, and Canada.
To evaluate the criteria discussed in Section SECREF1: (b) textual similarity, (c) fluency, and (d) sentiment preservation, we ask the annotators three questions:
Is snippet B a good paraphrase of snippet A?
$\circledcirc $ Yes $\circledcirc $ Somewhat yes $\circledcirc $ No
How natural does the text read?
$\circledcirc $ Very unnatural $\circledcirc $ Somewhat natural $\circledcirc $ Natural
What is the sentiment of the text?
$\circledcirc $ Positive $\circledcirc $ Negative $\circledcirc $ Cannot tell
For question 1, we display both the adversarial input and the original input, while for question 2 and 3 we present only the adversarial example. As an upper-bound, we also run a survey on question 2 and 3 for 50 random original samples.

Conclusion
We propose an evaluation framework for assessing the quality of adversarial examples in NLP, based on four criteria: (a) attacking performance, (b) textual similarity; (c) fluency; (d) label preservation. Our framework involves both automatic and human evaluation, and we test 5 benchmark methods and a novel auto-encoder approach. We found that the architecture of the target classifier is an important factor when it comes to attacking performance, e.g. CNNs are more vulnerable than LSTMs; dataset features such as length of text, training data size (for target classifiers) and input domains are also influencing factors that affect how difficulty it is to perform adversarial attack; and the predominant dataset (IMDB) used by most studies is comparatively easy for adversarial attack. Lastly, we also observe in our human evaluation that on shorter texts (Yelp) these methods produce adversarial examples that tend not to preserve their semantic content and have low readability. More importantly, these methods also “cheat” by simply flipping the sentiment in the adversarial examples, and this behaviour is evident on both datasets, suggesting they could be ineffective for adversarial training.

Table 1: Dataset statistics.

Table 2: Results based on automatic metrics. Top half (A) presents 3 different target classifiers evaluated on one dataset (yelp50); bottom half (B) tests 3 datasets using one classifier (BiLSTM+A). For ACPT, less negative values are better. Boldface indicates optimal performance in each column.

Figure 3: Human evaluation results.

Question: What other factors affect the performance?
{"answer": ["architecture of the classifier", "sentence length", " input domain"], "evidence": ["The core contribution of our paper is to introduce a systematic, rigorous evaluation framework to assess the quality of adversarial examples for NLP. We focus on sentiment classification as the target task, as it is a popular application that highlights the importance of criteria discussed above. We test a number of attacking methods and also propose an alternative approach (based on an auto-encoder) for generating adversarial examples. We learn that a number of factors can influence the performance of adversarial attacks, including architecture of the classifier, sentence length and input domain."]}