You will be shown sections from a scientific research paper, together with a question about the paper. Your task is to answer the question based on the contents of the paper. Your response should be a `json` object with two fields:

"answer": An array of strings extracted directly from the paper which, collectively, answer the question.
"evidence": An array of strings. Each should be an excerpt from the paper, in which one or more of the extracted answers can be found.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": ["BERT", "RoBERTa"],
  "evidence": ["In our experiments, we compare the performance of our model against BERT and RoBERTa."]
}

Do not include any text in your response other than the json.

If the question is unanswerable given the provided excerpts, respond with the single word "null".

Paper: Measuring Offensive Speech in Online Political Discourse

The Internet and online forums such as Reddit have become an increasingly popular medium for citizens to engage in political conversations. However, the online disinhibition effect resulting from the ability to use pseudonymous identities may manifest in the form of offensive speech, consequently making political discussions more aggressive and polarizing than they already are. Such environments may result in harassment and self-censorship from its targets. In this paper, we present preliminary results from a large-scale temporal measurement aimed at quantifying offensiveness in online political discussions. To enable our measurements, we develop and evaluate an offensive speech classifier. We then use this classifier to quantify and compare offensiveness in the political and general contexts. We perform our study using a database of over 168M Reddit comments made by over 7M pseudonyms between January 2015 and January 2017 -- a period covering several divisive political events including the 2016 US presidential elections.

Introduction
The apparent rise in political incivility has attracted substantial attention from scholars in recent years. These studies have largely focused on the extent to which politicians and elected officials are increasingly employing rhetoric that appears to violate norms of civility BIBREF0 , BIBREF1 . For the purposes of our work, we use the incidence of offensive rhetoric as a stand in for incivility. The 2016 US presidential election was an especially noteworthy case in this regard, particularly in terms of Donald Trump's campaign which frequently violated norms of civility both in how he spoke about broad groups in the public (such as Muslims, Mexicans, and African Americans) and the attacks he leveled at his opponents BIBREF2 . The consequences of incivility are thought to be crucial to the functioning of democracy since “public civility and interpersonal politeness sustain social harmony and allow people who disagree with one another to maintain ongoing relationships" BIBREF3 .
While political incivility appears to be on the rise among elites, it is less clear whether this is true among the mass public as well. Is political discourse particularly lacking in civility compared to discourse more generally? Does the incivility of mass political discourse respond to the dynamics of political campaigns? Addressing these questions has been difficult for political scientists because traditional tools for studying mass behavior, such as public opinion surveys, are ill-equipped to measure how citizens discuss politics with one another. Survey data does reveal that the public tends to perceive politics as becoming increasingly less civil during the course of a political campaign BIBREF4 . Yet, it is not clear whether these perceptions match the reality, particularly in terms of the types of discussions that citizens have with each other.
An additional question is how incivility is received by others. On one hand, violations of norms regarding offensive discourse may be policed by members of a community, rendering such speech ineffectual. On the other hand, offensive speech may be effective as a means for drawing attention to a particular argument. Indeed, there is evidence that increasing incivility in political speech results in higher levels of attention from the public BIBREF1 . During the 2016 campaign, the use of swearing in comments posted on Donald Trump's YouTube channel tended to result in additional responses that mimicked such swearing BIBREF5 . Thus, offensive speech in online fora may attract more attention from the community and lead to the spread of even more offensive speech in subsequent posts.
To address these questions regarding political incivility, we examine the use of offensive speech in political discussions housed on Reddit. Scholars tend to define uncivil discourse as “communication that violates the norms of politeness" BIBREF1 a definition that clearly includes offensive remarks. Reddit fora represent a “most likely" case for the study of offensive political speech due its strong free speech culture BIBREF6 and the ability of participants to use pseudonymous identities. That is, if political incivility in the public did increase during the 2016 campaign, this should be especially evident on fora such as Reddit. Tracking Reddit discussions throughout all of 2015 and 2016, we find that online political discussions became increasingly more offensive as the general election campaign intensified. By comparison, discussions on non-political subreddits did not become increasingly offensive during this period. Additionally, we find that the presence of offensive comments did not subside even three months after the election.

Datasets
Our study makes use of multiple datasets in order to identify and characterize trends in offensive speech.
The CrowdFlower hate speech dataset. The CrowdFlower hate speech dataset BIBREF7 contains 14.5K tweets, each receiving labels from at least three contributors. Contributors were allowed to classify each tweet into one of three classes: Not Offensive (NO), Offensive but not hateful (O), and Offensive and hateful (OH). Of the 14.5K tweets, only 37.6% had a decisive class – i.e., the same class was assigned by all contributors. For indecisive cases, the majority class was selected and a class confidence score (fraction of contributors that selected the majority class) was made available. Using this approach, 50.4%, 33.1%, and 16.5% of the tweets were categorized as NO, O, and OH, respectively. Since our goal is to identify any offensive speech (not just hate speech), we consolidate assigned classes into Offensive and Not Offensive by relabeling OH tweets as Offensive. We use this modified dataset to train, validate, and test our offensive speech classifier. To the best of our knowledge, this is the only dataset that provides offensive and not offensive annotations to a large dataset.
Offensive word lists. We also use two offensive word lists as auxiliary input to our classifier: (1) The Hatebase hate speech vocabulary BIBREF8 consisting of 1122 hateful words and (2) 422 offensive words banned from Google's What Do You Love project BIBREF9 .
Reddit comments dataset. Finally, after building our offensive speech classifier using the above datasets, we use it to classify comments made on Reddit. While the complete Reddit dataset contains 2B comments made between the period of January 2015 and January 2017, we only analyze only 168M. We select comments to be analyzed using the following process: (1) we exclude comments shorter than 10 characters in length, (2) we exclude comments made by [deleted] authors, and (3) we randomly sample and include 10% of all remaining comments. We categorize comments made in any of 21 popular political subreddits as political and the remainder as apolitical. Our final dataset contains 129M apolitical and 39M political comments. fig:comment-timeline shows the number of comments in our dataset that were made during each week included in our study. We see an increasing number of political comments per week starting in February 2016 – the start of the 2016 US presidential primaries.

Classification approach
At a high-level, our approach works as follows:
Build a word embedding. We construct a 100-dimensional word embedding using all comments from our complete Reddit dataset (2B comments).
Construct a hate vector. We construct a list of offensive and hateful words identified from external data and map them into a single vector within the high-dimensional word embedding.
Text transformation and classification. Finally, we transform text to be classified into scalars representing their distance from the constructed hate vector and use these as input to a Random Forest classifier.
Building a word embedding. At a high-level, a word embedding maps words into a high-dimensional continuous vector space in such a way that semantic similarities between words are maintained. This mapping is achieved by exploiting the distributional properties of words and their occurrences in the input corpus.
Rather than using an off-the-shelf word embedding (e.g., the GloVe embeddings BIBREF10 trained using public domain data sources such as Wikipedia and news articles), we construct a 100-dimensional embedding using the complete Reddit dataset (2B comments) as the input corpus. The constructed embedding consists of over 400M unique words (words occurring less than 25 times in the entire corpus are excluded) using the Word2Vec BIBREF11 implementation provided by the Gensim library BIBREF12 . Prior to constructing the embedding, we perform stop-word removal and lemmatize each word in the input corpus using the SpaCy NLP framework BIBREF13 . The main reason for building a custom embedding is to ensure that our embeddings capture semantics specific to the data being measured (Reddit) – e.g., while the word “karma” in the non-Reddit context may be associated with spirituality, it is associated with points (comment and submission scores) on Reddit.
Constructing a hate vector. We use two lists of words associated with hate BIBREF8 and offensive BIBREF9 speech to construct a hate vector in our word embedding. This is done by mapping each word in the list into the 100-dimensional embedding and computing the mean vector. This vector represents the average of all known offensive words. The main idea behind creating a hate vector is to capture the point (in our embedding) to which the most offensive observed comments are likely to be near. Although clustering our offensive word lists into similar groups and constructing multiple hate vectors – one for each cluster – results in marginally better accuracy for our classifier, we use this approach due to the fact that our classification cost grows linearly with the number of hate vectors – i.e., we need to perform INLINEFORM0 distance computations per hate vector to classify string INLINEFORM1 .
Transforming and classifying text. We first remove stop-words and perform lemmatization of each word in the text to be classified. We then obtain the vector representing each word in the text and compute its similarity to the constructed hate vector using the cosine similarity metric. A 0-vector is used to represent words in the text that are not present in the embedding. Finally, the maximum cosine similarity score is used to represent the comment. Equation EQREF7 shows the transformation function on a string INLINEFORM0 = INLINEFORM1 where INLINEFORM2 is the vector representing the INLINEFORM3 lemmatized non-stop-word, INLINEFORM4 is the cosine-similarity function, and INLINEFORM5 is the hate vector. DISPLAYFORM0 
In words, the numerical value assigned to a text is the cosine similarity between the hate vector and the vector representing the word (in the text) closest to the hate vector. This approach allows us to transform a string of text into a single numerical value that captures its semantic similarity to the most offensive comment. We use these scalars as input to a random forest classifier to perform classification into Offensive and Not Offensive classes. fig:reduced-dimension-classes shows the proximity of Offensive and Non Offensive comments to our constructed hate vector after using t-distributed Stochastic Neighbor Embedding (t-SNE) BIBREF14 to reduce our 100-dimension vector space into 2 dimensions.

Classifier evaluation
We now present results to (1) validate our choice of classifier and (2) demonstrate the impact of training/validation sample count on our classifiers performance.
Classifier selection methodology. To identify the most suitable classifier for classifying the scalars associated with each text, we perform evaluations using the stochastic gradient descent, naive bayes, decision tree, and random forest classifiers. For each classifier, we split the CrowdFlower hate speech dataset into a training/validation set (75%), and a holdout set (25%). We perform 10-fold cross-validation on the training/validation set to identify the best classifier model and parameters (using a grid search). Based on the results of this evaluation, we select a 100-estimator entropy-based splitting random forest model as our classifier. tab:classifiers shows the mean accuracy and F1-score for each evaluated classifier during the 10-fold cross-validation.
Real-world classifier performance. To evaluate real-world performance of our selected classifier (i.e., performance in the absence of model and parameter bias), we perform classification of the holdout set. On this set, our classifier had an accuracy and F1-score of 89.6% and 89.2%, respectively. These results show that in addition to superior accuracy during training and validation, our chosen classifier is also robust against over-fitting.
Impact of dataset quality and size. To understand how the performance of our chosen classifier model and parameters are impacted by: (1) the quality and consistency of manually assigned classes in the CrowdFlower dataset and (2) the size of the dataset, we re-evaluate the classifier while only considering tweets having a minimum confidence score and varying the size of the holdout set. Specifically, our experiments considered confidence thresholds of 0 (all tweets considered), .35 (only tweets where at least 35% of contributors agreed on a class were considered), and .70 (only tweets where at least 70% of the contributors agreed on a class were considered) and varied the holdout set sizes between 5% and 95% of all tweets meeting the confidence threshold set for the experiment.
The results illustrated in fig:classifier-performance show the performance of the classifier while evaluating the corresponding holdout set. We make several conclusions from these results:
Beyond a (fairly low) threshold, the size of the training and validation set has little impact on classifier performance. We see that the accuracy, precision, and recall have, at best, marginal improvements with holdout set sizes smaller than 60%. This implies that the CrowdFlower dataset is sufficient for building an offensive speech classifier.
Quality of manual labeling has a significant impact on the accuracy and precision of the classifier. Using only tweets which had at least 70% of contributors agreeing on a class resulted in between 5-7% higher accuracy and up to 5% higher precision.
Our classifier achieves precision of over 95% and recall of over 85% when considering only high confidence samples. This implies that the classifier is more likely to underestimate the presence of offensive speech – i.e., our results likely provide a lower-bound on the quantity of observed offensive speech.

Figure 1: Number of analyzed political and apolitical comments belonging to each week between January 2015 and January 2017.

Figure 3: Classifier performance on holdout sets while varying holdout set sizes and minimum confidence thresholds.

Figure 4: Fraction of offensive comments identified in political and all subreddits.

Figure 6: CDF of the fraction of each authors comments that were identified as offensive. Green, orange, and red dots are used to represent authors with <5, 5-15, and >15 total comments, respectively. The legend provides a breakdown per quartile.

Figure 7: Distribution of the fraction of offensive comments observed in each subreddit category. Only subreddits with over 1000 comments are considered.

Figure 8: Flow of offensive authors. An edge between two subreddits indicates that authors made offensive comments in the source subreddit before the first time they made offensive comments in the destination subreddit. Darker and thicker edges indicate larger flow sizes (only flows ≥ 200 authors are shown).

Question: What was the baseline?
{"answer": ["stochastic gradient descent, naive bayes, decision tree"], "evidence": ["Classifier selection methodology. To identify the most suitable classifier for classifying the scalars associated with each text, we perform evaluations using the stochastic gradient descent, naive bayes, decision tree, and random forest classifiers. For each classifier, we split the CrowdFlower hate speech dataset into a training/validation set (75%), and a holdout set (25%). We perform 10-fold cross-validation on the training/validation set to identify the best classifier model and parameters (using a grid search). Based on the results of this evaluation, we select a 100-estimator entropy-based splitting random forest model as our classifier. tab:classifiers shows the mean accuracy and F1-score for each evaluated classifier during the 10-fold cross-validation."]}