You will be shown sections from a scientific research paper, together with a question about the paper. Your task is to answer the question based on the contents of the paper. Your response should be a `json` object with two fields:

"answer": An array of strings extracted directly from the paper which, collectively, answer the question.
"evidence": An array of strings. Each should be an excerpt from the paper, in which one or more of the extracted answers can be found.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": ["BERT", "RoBERTa"],
  "evidence": ["In our experiments, we compare the performance of our model against BERT and RoBERTa."]
}

Do not include any text in your response other than the json.

If the question is unanswerable given the provided excerpts, respond with the single word "null".

Paper: Distinguishing Antonyms and Synonyms in a Pattern-based Neural Network

Distinguishing between antonyms and synonyms is a key task to achieve high performance in NLP systems. While they are notoriously difficult to distinguish by distributional co-occurrence models, pattern-based methods have proven effective to differentiate between the relations. In this paper, we present a novel neural network model AntSynNET that exploits lexico-syntactic patterns from syntactic parse trees. In addition to the lexical and syntactic information, we successfully integrate the distance between the related words along the syntactic path as a new pattern feature. The results from classification experiments show that AntSynNET improves the performance over prior pattern-based methods.

Introduction
Antonymy and synonymy represent lexical semantic relations that are central to the organization of the mental lexicon BIBREF0 . While antonymy is defined as the oppositeness between words, synonymy refers to words that are similar in meaning BIBREF1 , BIBREF2 . From a computational point of view, distinguishing between antonymy and synonymy is important for NLP applications such as Machine Translation and Textual Entailment, which go beyond a general notion of semantic relatedness and require to identify specific semantic relations. However, due to interchangeable substitution, antonyms and synonyms often occur in similar contexts, which makes it challenging to automatically distinguish between them.
Two families of approaches to differentiate between antonyms and synonyms are predominent in NLP. Both make use of distributional vector representations, relying on the distributional hypothesis BIBREF3 , BIBREF4 , that words with similar distributions have related meanings: co-occurrence models and pattern-based models. These distributional semantic models (DSMs) offer a means to represent meaning vectors of words or word pairs, and to determine their semantic relatedness BIBREF5 .
In co-occurrence models, each word is represented by a weighted feature vector, where features typically correspond to words that co-occur in particular contexts. When using word embeddings, these models rely on neural methods to represent words as low-dimensional vectors. To create the word embeddings, the models either make use of neural-based techniques, such as the skip-gram model BIBREF6 , or use matrix factorization BIBREF7 that builds word embeddings by factorizing word-context co-occurrence matrices. In comparison to standard co-occurrence vector representations, word embeddings address the problematic sparsity of word vectors and have achieved impressive results in many NLP tasks such as word similarity (e.g., Pennington:14), relation classification (e.g., ThangVu:16), and antonym-synonym distinction (e.g., Nguyen:16).
In pattern-based models, vector representations make use of lexico-syntactic surface patterns to distinguish between the relations of word pairs. For example, Justeson/Katz:91 suggested that adjectival opposites co-occur with each other in specific linear sequences, such as between X and Y. Hearst:92 determined surface patterns, e.g., X such as Y, to identify nominal hypernyms. Lin2003 proposed two textual patterns indicating semantic incompatibility, from X to Y and either X or Y, to distinguish opposites from semantically similar words. Roth2014 proposed a method that combined patterns with discourse markers for classifying paradigmatic relations including antonymy, synonymy, and hypernymy. Recently, Schwartz:15 used two prominent patterns from Lin2003 to learn word embeddings that distinguished antonyms from similar words in determining degrees of similarity and word analogy.
In this paper, we present a novel pattern-based neural method AntSynNET to distinguish antonyms from synonyms. We hypothesize that antonymous word pairs co-occur with each other in lexico-syntactic patterns within a sentence more often than would be expected by synonymous pairs. This hypothesis is inspired by corpus-based studies on antonymy and synonymy. Among others, Charles1989 suggested that adjectival opposites co-occur in patterns; Fellbaum:95 stated that nominal and verbal opposites co-occur in the same sentence significantly more often than chance; Lin2003 argued that if two words appear in clear antonym patterns, they are unlikely to represent synonymous pair.
We start out by inducing patterns between X and Y from a large-scale web corpus, where X and Y represent two words of an antonym or synonym word pair, and the pattern is derived from the simple paths between X and Y in a syntactic parse tree. Each node in the simple path combines lexical and syntactic information; in addition, we suggest a novel feature for the patterns, i.e., the distance between the two words along the syntactic path. All pattern features are fed into a recurrent neural network with long short-term memory (LSTM) units BIBREF8 , which encode the patterns as vector representations. Afterwards, the vector representations of the patterns are used in a classifier to distinguish between antonyms and synonyms. The results from experiments show that AntSynNET improves the performance over prior pattern-based methods. Furthermore, the implementation of our models is made publicly available.
The remainder of this paper is organized as follows: In Section "Related Work" , we present previous work distinguishing antonyms and synonyms. Section "AntSynNET: LSTM-based Antonym-Synonym Distinction" describes our proposed AntSynNET model. We present the induction of the patterns (Section "Baseline Models" ), describe the recurrent neural network with long short-term memory units which is used to encode patterns within a vector representation (Section "Recurrent Neural Network with Long Short-Term Memory Units" ), and describe two models to classify antonyms and synonyms: the pure pattern-based model (Section UID11 ) and the combined model (Section UID13 ). After introducing two baselines in Section "Baseline Models" , we describe our dataset, experimental settings, results of our methods, the effects of the newly proposed distance feature, and the effects of the various types of word embeddings. Section "Conclusion" concludes the paper.

AntSynNET: LSTM-based Antonym-Synonym Distinction
In this section, we describe the AntSynNET model, using a pattern-based LSTM for distinguishing antonyms from synonyms. We first present the induction of patterns from a parsed corpus (Section "Baseline Models" ). Section "Recurrent Neural Network with Long Short-Term Memory Units" then describes how we utilize the recurrent neural network with long short-term memory units to encode the patterns as vector representation. Finally, we present the AntSynNET model and two approaches to classify antonyms and synonyms (Section "The Proposed AntSynNET Model" ).

Baseline Models
To compare AntSynNET with baseline models for pattern-based classification of antonyms and synonyms, we introduce two pattern-based baseline methods: the distributional method (Section "Distributional Baseline" ), and the distributed method (Section "Distributed Baseline" ).

Dataset
For training the models, neural networks require a large amount of training data. We use the existing large-scale antonym and synonym pairs previously used by Nguyen:16. Originally, the data pairs were collected from WordNet BIBREF9 and Wordnik.
In order to induce patterns for the word pairs in the dataset, we identify the sentences in the corpus that contain the word pair. Thereafter, we extract all patterns for the word pair. We filter out all patterns which occur less than five times; and we only take into account word pairs that contain at least five patterns for training, validating and testing. For the proportion of positive and negative pairs, we keep a ratio of 1:1 positive (antonym) to negative (synonym) pairs in the dataset. In order to create the sets of training, testing and validation data, we perform random splitting with 70% train, 25% test, and 5% validation sets. The final dataset contains the number of word pairs according to word classes described in Table 1 . Moreover, Table 2 shows the average number of patterns for each word pair in our dataset.

Experimental Settings
We use the English Wikipedia dump from June 2016 as the corpus resource for our methods and baselines. For parsing the corpus, we rely on spaCy. For the lemma embeddings, we rely on the word embeddings of the dLCE model BIBREF10 which is the state-of-the-art vector representation for distinguishing antonyms from synonyms. We re-implemented this cutting-edge model on Wikipedia with 100 dimensions, and then make use of the dLCE word embeddings for initialization the lemma embeddings. The embeddings of POS tags, dependency labels, distance labels, and out-of-vocabulary lemmas are initialized randomly. The number of dimensions is set to 10 for the embeddings of POS tags, dependency labels and distance labels. We use the validation sets to tune the number of dimensions for these labels. For optimization, we rely on the cross-entropy loss function and Stochastic Gradient Descent with the Adadelta update rule BIBREF11 . For training, we use the Theano framework BIBREF12 . Regularization is applied by a dropout of 0.5 on each of component's embeddings (dropout rate is tuned on the validation set). We train the models with 40 epochs and update all embeddings during training.

Overall Results
Table 3 shows the significant performance of our models in comparison to the baselines. Concerning adjectives, the two proposed models significantly outperform the two baselines: The performance of the baselines is around .72 for $F_1$ , and the corresponding results for the combined AntSynNET model achieve an improvement of $>$ .06. Regarding nouns, the improvement of the new methods is just .02 $F_1$ in comparison to the R&SiW baseline, but we achieve a much better performance in comparison to the SP baseline, an increase of .37 $F_1$ . Regarding verbs, we do not outperform the more advanced R&SiW baseline in terms of the $F_1$ score, but we obtain higher recall scores. In comparison to the SP baseline, our models still show a clear $F_1$ improvement.
Overall, our proposed models achieve comparatively high recall scores compared to the two baselines. This strengthens our hypothesis that there is a higher possibility for the co-occurrence of antonymous pairs in patterns over synonymous pairs within a sentence. Because, when the proposed models obtain high recall scores, the models are able to retrieve most relevant information (antonymous pairs) corresponding to the patterns. Regarding the low precision in the two proposed models, we sampled randomly 5 pairs in each population: true positive, true negative, false positive, false negative. We then compared the overlap of patterns for the true predictions (true positive pairs and true negative pairs) and the false predictions (false positive pairs and false negative pairs). We found out that there is no overlap between patterns of true predictions; and the number overlap between patterns of false predictions is 2, 2, and 4 patterns for noun, adjective, and verb classes, respectively. This shows that the low precision of our models stems from the patterns which represent both antonymous and synonymous pairs.

Effect of Word Embeddings
Our methods rely on the word embeddings of the dLCE model, state-of-the-art word embeddings for antonym-synonym distinction. Yet, the word embeddings of the dLCE model, i.e., supervised word embeddings, represent information collected from lexical resources. In order to evaluate the effect of these word embeddings on the performance of our models, we replace them by the pre-trained GloVe word embeddings with 100 dimensions, and compare the effects of the GloVe word embeddings and the dLCE word embeddings on the performance of the two proposed models.
Table 5 illustrates the performance of our two models on all word classes. The table shows that the dLCE word embeddings are better than the pre-trained GloVe word embeddings, by around .01 $F_1$ for the pattern-based AntSynNET model and the combined AntSynNET model regarding adjective and verb pairs. Regarding noun pairs, the improvements of the dLCE word embeddings over pre-trained GloVe word embeddings achieve around .01 and .04 $F_1$ for the pattern-based model and the combined model, respectively.

Acknowledgements
We would like to thank Michael Roth for helping us to compute the results of the R&SiW model on our dataset.
The research was supported by the Ministry of Education and Training of the Socialist Republic of Vietnam (Scholarship 977/QD-BGDDT; Kim-Anh Nguyen), the DFG Collaborative Research Centre SFB 732 (Kim-Anh Nguyen, Ngoc Thang Vu), and the DFG Heisenberg Fellowship SCHU-2580/1 (Sabine Schulte im Walde).

Table 2: Average number of patterns per word pair across word classes.

Table 3: Performance of the AntSynNET models in comparison to the baseline models.

Table 5: Comparing pre-trained GloVe and dLCE word embeddings.

Question: What dataset do they use to evaluate their method?
{"answer": ["antonym and synonym pairs", "collected from WordNet BIBREF9 and Wordnik"], "evidence": ["For training the models, neural networks require a large amount of training data. We use the existing large-scale antonym and synonym pairs previously used by Nguyen:16. Originally, the data pairs were collected from WordNet BIBREF9 and Wordnik."]}