You will be shown sections from a scientific research paper, together with a question about the paper. Your task is to answer the question based on the contents of the paper. Your response should be a `json` object with two fields:

"answer": An array of strings extracted directly from the paper which, collectively, answer the question.
"evidence": An array of strings. Each should be an excerpt from the paper, in which one or more of the extracted answers can be found.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": ["BERT", "RoBERTa"],
  "evidence": ["In our experiments, we compare the performance of our model against BERT and RoBERTa."]
}

Do not include any text in your response other than the json.

If the question is unanswerable given the provided excerpts, respond with the single word "null".

Paper: Arabic Offensive Language on Twitter: Analysis and Experiments

Detecting offensive language on Twitter has many applications ranging from detecting/predicting bullying to measuring polarization. In this paper, we focus on building effective Arabic offensive tweet detection. We introduce a method for building an offensive dataset that is not biased by topic, dialect, or target. We produce the largest Arabic dataset to date with special tags for vulgarity and hate speech. Next, we analyze the dataset to determine which topics, dialects, and gender are most associated with offensive tweets and how Arabic speakers use offensive language. Lastly, we conduct a large battery of experiments to produce strong results (F1 = 79.7) on the dataset using Support Vector Machine techniques.

Introduction
Disclaimer: Due to the nature of the paper, some examples contain highly offensive language and hate speech. They don't reflect the views of the authors in any way, and the point of the paper is to help fight such speech. Much recent interest has focused on the detection of offensive language and hate speech in online social media. Such language is often associated with undesirable online behaviors such as trolling, cyberbullying, online extremism, political polarization, and propaganda. Thus, offensive language detection is instrumental for a variety of application such as: quantifying polarization BIBREF0, BIBREF1, trolls and propaganda account detection BIBREF2, detecting the likelihood of hate crimes BIBREF3; and predicting conflict BIBREF4. In this paper, we describe our methodology for building a large dataset of Arabic offensive tweets. Given that roughly 1-2% of all Arabic tweets are offensive BIBREF5, targeted annotation is essential for efficiently building a large dataset. Since our methodology does not use a seed list of offensive words, it is not biased by topic, target, or dialect. Using our methodology, we tagged 10,000 Arabic tweet dataset for offensiveness, where offensive tweets account for roughly 19% of the tweets. Further, we labeled tweets as vulgar or hate speech. To date, this is the largest available dataset, which we plan to make publicly available along with annotation guidelines. We use this dataset to characterize Arabic offensive language to ascertain the topics, dialects, and users' gender that are most associated with the use of offensive language. Though we suspect that there are common features that span different languages and cultures, some characteristics of Arabic offensive language is language and culture specific. Thus, we conduct a thorough analysis of how Arabic users use offensive language. Next, we use the dataset to train strong Arabic offensive language classifiers using state-of-the-art representations and classification techniques. Specifically, we experiment with static and contextualized embeddings for representation along with a variety of classifiers such as a deep neural network classifier and Support Vector Machine (SVM).
The contributions of this paper are as follows:
We built the largest Arabic offensive language dataset to date that includes special tags for vulgar language and hate speech. We describe the methodology for building it along with annotation guidelines.
We performed thorough analysis of the dataset and described the peculiarities of Arabic offensive language.
We experimented with Support Vector Machine classifiers on character and word ngrams classification techniques to provide strong results on Arabic offensive language classification.

Related Work
Many recent papers have focused on the detection of offensive language, including hate speech BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10, BIBREF11, BIBREF12, BIBREF13. Offensive language can be categorized as: Vulgar, which include explicit and rude sexual references, Pornographic, and Hateful, which includes offensive remarks concerning people’s race, religion, country, etc. BIBREF14. Prior works have concentrated on building annotated corpora and training classification models. Concerning corpora, hatespeechdata.com attempts to maintain an updated list of hate speech corpora for multiple languages including Arabic and English. Further, SemEval 2019 ran an evaluation task targeted at detecting offensive language, which focused exclusively on English BIBREF15. As for classification models, most studies used supervised classification at either word level BIBREF10, character sequence level BIBREF11, and word embeddings BIBREF9. The studies used different classification techniques including using Naïve Bayes BIBREF10, SVM BIBREF11, and deep learning BIBREF6, BIBREF7, BIBREF12 classification. The accuracy of the aforementioned system ranged between 76% and 90%. Earlier work looked at the use of sentiment words as features as well as contextual features BIBREF13.
The work on Arabic offensive language detection is relatively nascent BIBREF16, BIBREF17, BIBREF18, BIBREF19, BIBREF5. Mubarak et al. mubarak2017abusive suggested that certain users are more likely to use offensive languages than others, and they used this insight to build a list of offensive Arabic words and they constructed a labeled set of 1,100 tweets. Abozinadah et al. abozinadah2017detecting used supervised classification based on a variety of features including user profile features, textual features, and network features. They reported an accuracy of nearly 90%. Alakrot et al. alakrot2018towards used supervised classification based on word unigrams and n-grams to detect offensive language in YouTube comments. They improved classification with stemming and achieved a precision of 88%. Albadi et al. albadi2018they focused on detecting religious hate speech using a recurrent neural network. Further; Schmidt and Wiegand schmidt-wiegand-2017-survey surveyed major works on hate speech detection; Fortuna and Nunes Fortuna2018Survey provided a comprehensive survey for techniques and works done in the area between 2004 and 2017.
Arabic is a morphologically rich language with a standard variety, namely Modern Standard Arabic (MSA) and is typically used in formal communication, and many dialectal varieties that differ from MSA in lexical selection, morphology, and syntactic structures. For MSA, words are typically derived from a set of thousands of roots by fitting a root into a stem template and the resulting stem may accept a variety of prefixes and suffixes such as coordinating conjunctions and pronouns. Though word segmentation (or stemming) is quite accurate for MSA BIBREF20, with accuracy approaching 99%, dialectal segmentation is not sufficiently reliable, with accuracy ranging between 91-95% for different dialects BIBREF21. Since dialectal Arabic is ubiquitous in Arabic tweets and many tweets have creative spellings of words, recent work on Arabic offensive language detection used character-level models BIBREF5.

Data Collection ::: Annotating Tweets ::: HATE SPEECH (HS):
Hate speech tweets, a subset of offensive tweets containing offensive language targeting group based on common characteristics such as: Race, ex: يا زنجي> (“yA znjy” – “O negro”); Ethnicity, ex. الفرس الأنجاس> (“Alfrs AlAnjAs” – “Impure Persians”); Group or party, ex: أبوك شيوعي> (“Abwk $ywEy” – “your father is communist”); and Religion, ex: دينك القذر> (“dynk Alq*r” – “your filthy religion”).

Experiments ::: Representations ::: Lexical Features
Since offensive words typically have a negative polarity, we wanted to test the effectiveness of using a polarity lexicon in detecting offensive tweets. For the lexicon, we used NileULex BIBREF26, which is an Arabic polarity lexicon containing 3,279 MSA and 2,674 Egyptian terms, out of which 4,256 are negative and 1,697 are positive. We used the counts of terms with positive polarity and terms with negative polarity in tweets as features.

Experiments ::: Representations ::: Deep Contextualized Embeddings
We also experimented with pre-trained contextualized embeddings with fine-tuning for down-stream tasks. Recently, deep contextualized language models such as BERT (Bidirectional Encoder Representations from Transformers) BIBREF31, UMLFIT BIBREF32, and OpenAI GPT BIBREF33, to name but a few, have achieved ground-breaking results in many NLP classification and language understanding tasks. In this paper, we used BERTbase-multilingual (hereafter as simply BERT) fine-tuning method to classify Arabic offensive language on Twitter as it eliminates the need of heavily engineered task-specific architectures. Although Robustly Optimized BERT (RoBERTa) embeddings perform better than (BERTlarge) on GLUE BIBREF34, RACE BIBREF35, and SQuAD BIBREF36 tasks, pre-trained multilingual RoBERTa models are not available. BERT is pre-trained on Wikipedia text from 104 languages and comes with hundreds of millions of parameters. It contains an encoder with 12 Transformer blocks, hidden size of 768, and 12 self-attention heads. Though the training data for the BERT embeddings don't match our genre, these embedding use BP sub-word segments. Following devlin-2019-bert, the classification consists of introducing a dense layer over the final hidden state $h$ corresponding to first token of the sequence, [CLS], adding a softmax activation on the top of BERT to predict the probability of the $l$ label:
where $W$ is the task-specific weight matrix. During fine-tuning, all BERT parameters together with $W$ are optimized end-to-end to maximize the log-probability of the correct labels.

Experiments ::: Classification Models
We explored different classifiers. When using lexical features and pre-trained static embeddings, we primarily used an SVM classifier with a radial basis function kernel. Only when using the Mazajak embeddings, we experimented with other classifiers such as AdaBoost and Logistic regression. We did so to show that the SVM classifier was indeed the best of the bunch, and we picked the Mazajak embeddings because they yielded the best results among all static embeddings. We used the Scikit Learn implementations of all the classifiers such as libsvm for the SVM classifier. We also experimented with fastText, which trained embeddings on our data. When using contextualized embeddings, we fine-tuned BERT by adding a fully-connected dense layer followed by a softmax classifier, minimizing the binary cross-entropy loss function for the training data. For all experiments, we used the PyTorch implementation by HuggingFace as it provides pre-trained weights and vocabulary.

Experiments ::: Evaluation
For all of our experiments, we used 5-fold cross validation with identical folds for all experiments. Table TABREF27 reports on the results of using lexical features, static pre-trained embeddings with an SVM classifier, embeddings trained on our data with fastText classifier, and BERT over a dense layer with softmax activation. As the results show, using Mazajak/SVM yielded the best results overall with large improvements in precision over using BERT. We suspect that the Mazajak/SVM combination performed better than the BERT setup due to the fact that the Mazajak embeddings, though static, were trained on in-domain data, as opposed to BERT. Perhaps if BERT embeddings were trained on tweets, they might have outperformed all other setups. For completeness, we compared 7 other classifiers with SVM using Mazajak embeddings. As results in Table TABREF28 show, using SVM yielded the best results.

Conclusion and Future Work
In this paper we presented a systematic method for building an Arabic offensive language tweet dataset that does not favor specific dialects, topics, or genres. We developed detailed guidelines for tagging the tweets as clean or offensive, including special tags for vulgar tweets and hate speech. We tagged 10,000 tweets, which we plan to release publicly and would constitute the largest available Arabic offensive language dataset. We characterized the offensive tweets in the dataset to determine the topics that illicit such language, the dialects that are most often used, the common modes of offensiveness, and the gender distribution of their authors. We performed this breakdown for offensive tweets in general and for vulgar and hate speech tweets separately. We believe that this is the first detailed analysis of its kind. Lastly, we conducted a large battery of experiments on the dataset, using cross-validation, to establish a strong system for Arabic offensive language detection. We showed that using static embeddings produced a competitive results on the dataset.
For future work, we plan to pursue several directions. First, we want explore target specific offensive language, where attacks against an entity or a group may employ certain expressions that are only offensive within the context of that target and completely innocuous otherwise. Second, we plan to examine the effectiveness of cross dialectal and cross lingual learning of offensive language.

Table 1: Distribution of offensive and clean tweets.

Figure 1: Topic distribution for offensive language and its sub-categories

Figure 3: Gender distribution for offensive language and its sub-categories

Figure 2: Dialect distribution for offensive language and its sub-categories

Figure 5: Tag cloud for words with top valence score among offensive class

Table 2: Classification performance with different features and models.

Question: In what way is the offensive dataset not biased by topic, dialect or target?
{"answer": ["our methodology does not use a seed list of offensive words"], "evidence": ["Disclaimer: Due to the nature of the paper, some examples contain highly offensive language and hate speech. They don\u0027t reflect the views of the authors in any way, and the point of the paper is to help fight such speech. Much recent interest has focused on the detection of offensive language and hate speech in online social media. Such language is often associated with undesirable online behaviors such as trolling, cyberbullying, online extremism, political polarization, and propaganda. Thus, offensive language detection is instrumental for a variety of application such as: quantifying polarization BIBREF0, BIBREF1, trolls and propaganda account detection BIBREF2, detecting the likelihood of hate crimes BIBREF3; and predicting conflict BIBREF4. In this paper, we describe our methodology for building a large dataset of Arabic offensive tweets. Given that roughly 1-2% of all Arabic tweets are offensive BIBREF5, targeted annotation is essential for efficiently building a large dataset. Since our methodology does not use a seed list of offensive words, it is not biased by topic, target, or dialect. Using our methodology, we tagged 10,000 Arabic tweet dataset for offensiveness, where offensive tweets account for roughly 19% of the tweets. Further, we labeled tweets as vulgar or hate speech. To date, this is the largest available dataset, which we plan to make publicly available along with annotation guidelines. We use this dataset to characterize Arabic offensive language to ascertain the topics, dialects, and users\u0027 gender that are most associated with the use of offensive language. Though we suspect that there are common features that span different languages and cultures, some characteristics of Arabic offensive language is language and culture specific. Thus, we conduct a thorough analysis of how Arabic users use offensive language. Next, we use the dataset to train strong Arabic offensive language classifiers using state-of-the-art representations and classification techniques. Specifically, we experiment with static and contextualized embeddings for representation along with a variety of classifiers such as a deep neural network classifier and Support Vector Machine (SVM)."]}