You will be shown sections from a scientific research paper, together with a question about the paper. Your task is to answer the question based on the contents of the paper. Your response should be a `json` object with two fields:

"answer": An array of strings extracted directly from the paper which, collectively, answer the question.
"evidence": An array of strings. Each should be an excerpt from the paper, in which one or more of the extracted answers can be found.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": ["BERT", "RoBERTa"],
  "evidence": ["In our experiments, we compare the performance of our model against BERT and RoBERTa."]
}

Do not include any text in your response other than the json.

If the question is unanswerable given the provided excerpts, respond with the single word "null".

Paper: Measuring Offensive Speech in Online Political Discourse

The Internet and online forums such as Reddit have become an increasingly popular medium for citizens to engage in political conversations. However, the online disinhibition effect resulting from the ability to use pseudonymous identities may manifest in the form of offensive speech, consequently making political discussions more aggressive and polarizing than they already are. Such environments may result in harassment and self-censorship from its targets. In this paper, we present preliminary results from a large-scale temporal measurement aimed at quantifying offensiveness in online political discussions. To enable our measurements, we develop and evaluate an offensive speech classifier. We then use this classifier to quantify and compare offensiveness in the political and general contexts. We perform our study using a database of over 168M Reddit comments made by over 7M pseudonyms between January 2015 and January 2017 -- a period covering several divisive political events including the 2016 US presidential elections.

Introduction
The apparent rise in political incivility has attracted substantial attention from scholars in recent years. These studies have largely focused on the extent to which politicians and elected officials are increasingly employing rhetoric that appears to violate norms of civility BIBREF0 , BIBREF1 . For the purposes of our work, we use the incidence of offensive rhetoric as a stand in for incivility. The 2016 US presidential election was an especially noteworthy case in this regard, particularly in terms of Donald Trump's campaign which frequently violated norms of civility both in how he spoke about broad groups in the public (such as Muslims, Mexicans, and African Americans) and the attacks he leveled at his opponents BIBREF2 . The consequences of incivility are thought to be crucial to the functioning of democracy since “public civility and interpersonal politeness sustain social harmony and allow people who disagree with one another to maintain ongoing relationships" BIBREF3 .
While political incivility appears to be on the rise among elites, it is less clear whether this is true among the mass public as well. Is political discourse particularly lacking in civility compared to discourse more generally? Does the incivility of mass political discourse respond to the dynamics of political campaigns? Addressing these questions has been difficult for political scientists because traditional tools for studying mass behavior, such as public opinion surveys, are ill-equipped to measure how citizens discuss politics with one another. Survey data does reveal that the public tends to perceive politics as becoming increasingly less civil during the course of a political campaign BIBREF4 . Yet, it is not clear whether these perceptions match the reality, particularly in terms of the types of discussions that citizens have with each other.
An additional question is how incivility is received by others. On one hand, violations of norms regarding offensive discourse may be policed by members of a community, rendering such speech ineffectual. On the other hand, offensive speech may be effective as a means for drawing attention to a particular argument. Indeed, there is evidence that increasing incivility in political speech results in higher levels of attention from the public BIBREF1 . During the 2016 campaign, the use of swearing in comments posted on Donald Trump's YouTube channel tended to result in additional responses that mimicked such swearing BIBREF5 . Thus, offensive speech in online fora may attract more attention from the community and lead to the spread of even more offensive speech in subsequent posts.
To address these questions regarding political incivility, we examine the use of offensive speech in political discussions housed on Reddit. Scholars tend to define uncivil discourse as “communication that violates the norms of politeness" BIBREF1 a definition that clearly includes offensive remarks. Reddit fora represent a “most likely" case for the study of offensive political speech due its strong free speech culture BIBREF6 and the ability of participants to use pseudonymous identities. That is, if political incivility in the public did increase during the 2016 campaign, this should be especially evident on fora such as Reddit. Tracking Reddit discussions throughout all of 2015 and 2016, we find that online political discussions became increasingly more offensive as the general election campaign intensified. By comparison, discussions on non-political subreddits did not become increasingly offensive during this period. Additionally, we find that the presence of offensive comments did not subside even three months after the election.

Offensive Speech Classification
In order to identify offensive speech, we propose a fully automated technique that classifies comments into two classes: Offensive and Not Offensive.

Classifier evaluation
We now present results to (1) validate our choice of classifier and (2) demonstrate the impact of training/validation sample count on our classifiers performance.
Classifier selection methodology. To identify the most suitable classifier for classifying the scalars associated with each text, we perform evaluations using the stochastic gradient descent, naive bayes, decision tree, and random forest classifiers. For each classifier, we split the CrowdFlower hate speech dataset into a training/validation set (75%), and a holdout set (25%). We perform 10-fold cross-validation on the training/validation set to identify the best classifier model and parameters (using a grid search). Based on the results of this evaluation, we select a 100-estimator entropy-based splitting random forest model as our classifier. tab:classifiers shows the mean accuracy and F1-score for each evaluated classifier during the 10-fold cross-validation.
Real-world classifier performance. To evaluate real-world performance of our selected classifier (i.e., performance in the absence of model and parameter bias), we perform classification of the holdout set. On this set, our classifier had an accuracy and F1-score of 89.6% and 89.2%, respectively. These results show that in addition to superior accuracy during training and validation, our chosen classifier is also robust against over-fitting.
Impact of dataset quality and size. To understand how the performance of our chosen classifier model and parameters are impacted by: (1) the quality and consistency of manually assigned classes in the CrowdFlower dataset and (2) the size of the dataset, we re-evaluate the classifier while only considering tweets having a minimum confidence score and varying the size of the holdout set. Specifically, our experiments considered confidence thresholds of 0 (all tweets considered), .35 (only tweets where at least 35% of contributors agreed on a class were considered), and .70 (only tweets where at least 70% of the contributors agreed on a class were considered) and varied the holdout set sizes between 5% and 95% of all tweets meeting the confidence threshold set for the experiment.
The results illustrated in fig:classifier-performance show the performance of the classifier while evaluating the corresponding holdout set. We make several conclusions from these results:
Beyond a (fairly low) threshold, the size of the training and validation set has little impact on classifier performance. We see that the accuracy, precision, and recall have, at best, marginal improvements with holdout set sizes smaller than 60%. This implies that the CrowdFlower dataset is sufficient for building an offensive speech classifier.
Quality of manual labeling has a significant impact on the accuracy and precision of the classifier. Using only tweets which had at least 70% of contributors agreeing on a class resulted in between 5-7% higher accuracy and up to 5% higher precision.
Our classifier achieves precision of over 95% and recall of over 85% when considering only high confidence samples. This implies that the classifier is more likely to underestimate the presence of offensive speech – i.e., our results likely provide a lower-bound on the quantity of observed offensive speech.

Measurements
In this section we quantify and characterize offensiveness in the political and general contexts using our offensive speech classifier and the Reddit comments dataset which considers a random sample of comments made between January 2015 and January 2017.
Offensiveness over time. We find that on average 8.4% of all political comments are offensive compared to 7.8% of all apolitical comments. fig:offensive-speech-timeline illustrates the fraction of offensive political and apolitical comments made during each week in our study. We see that while the fraction of apolitical offensive comments has stayed steady, there has been an increase in the fraction of offensive political comments starting in July 2016. Notably, this increase is observed after the conclusion of the US presidential primaries and during the period of the Democratic and Republican National Conventions and does not reduce even after the conclusion of the US presidential elections held on November 8. Participants in political subreddits were 2.6% more likely to observe offensive comments prior to July 2016 but 14.9% more likely to observe offensive comments from July 2016 onwards.
Reactions to offensive comments. We use the comment score, roughly the difference between up-votes and down-votes received, as a proxy for understanding how users reacted to offensive comments. We find that comments that were offensive: (1) on average, had a higher score than non-offensive comments (average scores: 8.9 vs. 6.7) and (2) were better received when they were posted in the general context than in the political context (average scores: 8.6 vs. 9.0). To understand how peoples reactions to offensive comments evolved over time, fig:offensive-scores-timeline shows the average scores received by offensive comments over time. Again, we observe an increasing trend in average scores received by offensive and political comments after July 2016.
Characteristics of offensive authors. We now focus on understanding characteristics of authors of offensive comments. Specifically, we are interested in identifying the use of throwaway and troll accounts. For the purposes of this study, we characterize throwaway accounts as those with less than five total comments – i.e., accounts that are used to make a small number of comments. Similarly, we define troll accounts as those with over 15 comments of which over 75% are classified as offensive – i.e., accounts that are used to make a larger number of comments, of which a significant majority are offensive. We find that 93.7% of the accounts which have over 75% of their comments tagged as offensive are throwaways and 1.3% are trolls. Complete results are illustrated in fig:offensive-authors-cdf.
Characteristics of offensive communities. We breakdown subreddits by their category (default, political, and other) and identify the most and least offensive communities in each. fig:subreddit-cdf shows the distribution of the fraction of offensive comments in each category and tab:subreddit-breakdown shows the most and least offensive subreddits in the political and default categories (we exclude the “other” category due to the inappropriateness of their names). We find that less than 19% of all subreddits (that account for over 23% of all comments) have over 10% offensive comments. Further, several default and political subreddits fall in this category, including r/the INLINEFORM0 donald – the most offensive political subreddit and the subreddit dedicated to the US President.
Flow of offensive authors. Finally, we uncover patterns in the movement of offensive authors between communities. In fig:offensive-flow we show the communities in which large number of authors of offensive content on the r/politics subreddit had previously made offensive comments (we refer to these communities as sources). Unsurprisingly, the most popular sources belonged to the default subreddits (e.g., r/worldnews, r/wtf, r/videos, r/askreddit, and r/news). We find that several other political subreddits also serve as large sources of offensive authors. In fact, the subreddits dedicated to the three most popular US presidential candidates – r/the INLINEFORM0 donald, r/sandersforpresident, and r/hillaryclinton rank in the top three. Finally, outside of the default and political subreddits, we find that r/nfl, r/conspiracy, r/dota2, r/reactiongifs, r/blackpeopletwitter, and r/imgoingtohellforthis were the largest sources of offensive political authors.

Figure 2: Proximity of offensive and non-offensive comments to the hate vector. Dimension reduction is performed using t-SNE.

Figure 4: Fraction of offensive comments identified in political and all subreddits.

Figure 6: CDF of the fraction of each authors comments that were identified as offensive. Green, orange, and red dots are used to represent authors with <5, 5-15, and >15 total comments, respectively. The legend provides a breakdown per quartile.

Figure 5: Average scores of offensive comments identified in political and all subreddits.

Figure 8: Flow of offensive authors. An edge between two subreddits indicates that authors made offensive comments in the source subreddit before the first time they made offensive comments in the destination subreddit. Darker and thicker edges indicate larger flow sizes (only flows ≥ 200 authors are shown).

Question: What other political events are included in the database?
{"answer": ["US presidential primaries", "Democratic and Republican National Conventions"], "evidence": ["Offensiveness over time. We find that on average 8.4% of all political comments are offensive compared to 7.8% of all apolitical comments. fig:offensive-speech-timeline illustrates the fraction of offensive political and apolitical comments made during each week in our study. We see that while the fraction of apolitical offensive comments has stayed steady, there has been an increase in the fraction of offensive political comments starting in July 2016. Notably, this increase is observed after the conclusion of the US presidential primaries and during the period of the Democratic and Republican National Conventions and does not reduce even after the conclusion of the US presidential elections held on November 8. Participants in political subreddits were 2.6% more likely to observe offensive comments prior to July 2016 but 14.9% more likely to observe offensive comments from July 2016 onwards."]}