You will be shown sections from a scientific research paper, together with a question about the paper. Paragraphs in the paper are separated by newlines. Your task is to answer the question based on the contents of the paper.

Paper:
----------------------------------------
Arabic Offensive Language on Twitter: Analysis and Experiments

Detecting offensive language on Twitter has many applications ranging from detecting/predicting bullying to measuring polarization. In this paper, we focus on building effective Arabic offensive tweet detection. We introduce a method for building an offensive dataset that is not biased by topic, dialect, or target. We produce the largest Arabic dataset to date with special tags for vulgarity and hate speech. Next, we analyze the dataset to determine which topics, dialects, and gender are most associated with offensive tweets and how Arabic speakers use offensive language. Lastly, we conduct a large battery of experiments to produce strong results (F1 = 79.7) on the dataset using Support Vector Machine techniques.

Related Work
Many recent papers have focused on the detection of offensive language, including hate speech BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10, BIBREF11, BIBREF12, BIBREF13. Offensive language can be categorized as: Vulgar, which include explicit and rude sexual references, Pornographic, and Hateful, which includes offensive remarks concerning people’s race, religion, country, etc. BIBREF14. Prior works have concentrated on building annotated corpora and training classification models. Concerning corpora, hatespeechdata.com attempts to maintain an updated list of hate speech corpora for multiple languages including Arabic and English. Further, SemEval 2019 ran an evaluation task targeted at detecting offensive language, which focused exclusively on English BIBREF15. As for classification models, most studies used supervised classification at either word level BIBREF10, character sequence level BIBREF11, and word embeddings BIBREF9. The studies used different classification techniques including using Naïve Bayes BIBREF10, SVM BIBREF11, and deep learning BIBREF6, BIBREF7, BIBREF12 classification. The accuracy of the aforementioned system ranged between 76% and 90%. Earlier work looked at the use of sentiment words as features as well as contextual features BIBREF13.
The work on Arabic offensive language detection is relatively nascent BIBREF16, BIBREF17, BIBREF18, BIBREF19, BIBREF5. Mubarak et al. mubarak2017abusive suggested that certain users are more likely to use offensive languages than others, and they used this insight to build a list of offensive Arabic words and they constructed a labeled set of 1,100 tweets. Abozinadah et al. abozinadah2017detecting used supervised classification based on a variety of features including user profile features, textual features, and network features. They reported an accuracy of nearly 90%. Alakrot et al. alakrot2018towards used supervised classification based on word unigrams and n-grams to detect offensive language in YouTube comments. They improved classification with stemming and achieved a precision of 88%. Albadi et al. albadi2018they focused on detecting religious hate speech using a recurrent neural network. Further; Schmidt and Wiegand schmidt-wiegand-2017-survey surveyed major works on hate speech detection; Fortuna and Nunes Fortuna2018Survey provided a comprehensive survey for techniques and works done in the area between 2004 and 2017.
Arabic is a morphologically rich language with a standard variety, namely Modern Standard Arabic (MSA) and is typically used in formal communication, and many dialectal varieties that differ from MSA in lexical selection, morphology, and syntactic structures. For MSA, words are typically derived from a set of thousands of roots by fitting a root into a stem template and the resulting stem may accept a variety of prefixes and suffixes such as coordinating conjunctions and pronouns. Though word segmentation (or stemming) is quite accurate for MSA BIBREF20, with accuracy approaching 99%, dialectal segmentation is not sufficiently reliable, with accuracy ranging between 91-95% for different dialects BIBREF21. Since dialectal Arabic is ubiquitous in Arabic tweets and many tweets have creative spellings of words, recent work on Arabic offensive language detection used character-level models BIBREF5.

Data Collection ::: Annotating Tweets
We developed the annotation guidelines jointly with an experienced annotator, who is a native Arabic speaker with a good knowledge of various Arabic dialects. We made sure that our guidelines were compatible with those of OffensEval2019. The annotator carried out all annotation. Tweets were given one or more of the following four labels: offensive, vulgar, hate speech, or clean. Since the offensive label covers both vulgar and hate speech and vulgarity and hate speech are not mutually exclusive, a tweet can be just offensive or offensive and vulgar and/or hate speech. The annotation adhered to the following guidelines:

Data Collection ::: Annotating Tweets ::: CLEAN (CLN):
Clean tweets do not contain vulgar or offensive language. We noticed that some tweets have some offensive words, but the whole tweet should not be considered as offensive due to the intention of users. This suggests that normal string match without considering contexts will fail in some cases. Examples of such ambiguous cases include: Humor, ex: يا عدوة الفرحة ههه> (“yA Edwp AlfrHp hhh” – “O enemy of happiness hahaha”); Advice, ex: لا تقل لصاحبك يا خنزير> (“lA tql lSAHbk yA xnzyr” – “don't say to your friend: You are a pig”); Condition, ex: إذا عارضتهم يقولون يا عميل> (“A*A EArDthm yqwlwn yA Emyl” – “if you disagree with them they will say: You are an agent”); Condemnation, ex: لماذا نسب بقول: يا بقرة؟> (“lmA*A nsb bqwl: yA bqrp?” – “Why do we insult others by saying: O cow?”); Self offense, ex: تعبت من لساني القذر> (“tEbt mn lsAny Alq*r” – “I am tired of my dirty tongue”); Non-human target, ex: يا بنت المجنونة يا كورة> (“yA bnt Almjnwnp yA kwrp” – “O daughter of the crazy one O football”); and Quotation from a movies or a story, ex: تاني يا زكي! تاني يا فاشل> (“tAny yA zky! tAny yA fA$l” – “again O Zaky! again O loser”). For other ambiguous cases, the annotator searched Twitter to find how actual users used expressions.
Table TABREF11 shows the distribution of the annotated tweets. There are 1,915 offensive tweets, including 225 vulgar tweet and 506 hate speech tweets, and 8,085 clean tweets. To validate the quality of annotation, a random sample of 100 tweets from the data, containing 50 offensive and 50 clean tweets, was given to additional three annotators. We calculated the Inter-Annotator Agreement between the annotators using Fleiss’s Kappa coefficient BIBREF23. The Kappa score was 0.92 indicating high quality annotation and agreement.

Experiments ::: Representations ::: Lexical Features
Since offensive words typically have a negative polarity, we wanted to test the effectiveness of using a polarity lexicon in detecting offensive tweets. For the lexicon, we used NileULex BIBREF26, which is an Arabic polarity lexicon containing 3,279 MSA and 2,674 Egyptian terms, out of which 4,256 are negative and 1,697 are positive. We used the counts of terms with positive polarity and terms with negative polarity in tweets as features.

Experiments ::: Representations ::: Static Embeddings
We experimented with various static embeddings that were pre-trained on different corpora with different vector dimensionality. We compared pre-trained embeddings to embeddings that were trained on our dataset. For pre-trained embeddings, we used: fastText Egyptian Arabic pre-trained embeddings BIBREF27 with vector dimensionality of 300; AraVec skip-gram embeddings BIBREF28, trained on 66.9M Arabic tweets with 100-dimensional vectors; and Mazajak skip-gram embeddings BIBREF29, trained on 250M Arabic tweets with 300-dimensional vectors.
Sentence embeddings were calculated by taking the mean of the embeddings of their tokens. The importance of testing a character level n-gram model like fastText lies in the agglutinative nature of the Arabic language. We trained a new fastText text classification model BIBREF30 on our dataset with vectors of 40 dimensions, 0.5 learning rate, 2$-$10 character n-grams as features, for 30 epochs. These hyper-parameters were tuned using a 5-fold cross-validated grid-search.

Experiments ::: Evaluation
For all of our experiments, we used 5-fold cross validation with identical folds for all experiments. Table TABREF27 reports on the results of using lexical features, static pre-trained embeddings with an SVM classifier, embeddings trained on our data with fastText classifier, and BERT over a dense layer with softmax activation. As the results show, using Mazajak/SVM yielded the best results overall with large improvements in precision over using BERT. We suspect that the Mazajak/SVM combination performed better than the BERT setup due to the fact that the Mazajak embeddings, though static, were trained on in-domain data, as opposed to BERT. Perhaps if BERT embeddings were trained on tweets, they might have outperformed all other setups. For completeness, we compared 7 other classifiers with SVM using Mazajak embeddings. As results in Table TABREF28 show, using SVM yielded the best results.

Experiments ::: Error Analysis ::: False Positives
had four main types:
[leftmargin=*]
Gloating: ex. يا هبيده> (“yA hbydp” - “O you delusional”) referring to fans of rival sports team for thinking they could win.
Quoting: ex. لما حد يسب ويقول يا كلب> (“lmA Hd ysb wyqwl yA klb” – “when someone swears and says: O dog”).
Idioms: ex. يا فاطر رمضان يا خاسر دينك> (“yA fATr rmDAn yA xAsr dynk” – “o you who does not fast Ramadan, you who have lost your religion”), which is a colloquial idiom.
Implicit Sarcasm: ex. يا خاين انت عايز تشكك>
في حب الشعب للريس> (“yA KAyn Ant EAwz t$kk fy Hb Al$Eb llrys” – “O traitor, (you) want to question the love of the people to the president ”) where the author is mocking the president's popularity.

Figure 1: Topic distribution for offensive language and its sub-categories

Figure 3: Gender distribution for offensive language and its sub-categories

Figure 4: Distribution of Hate Speech Types. Note: A tweet may have more than one type.

Table 3: Performance of different classification models on Mazajak embeddings.
----------------------------------------

Question: How many annotators tagged each tweet?

To answer the question, format your response as a `json` object with two fields:

"answer": A string providing a succinct answer to the question, in your own words.
"evidence": An array of strings. Each entry should be a full paragraph from the paper. Together, the evidence should serve as a justification for the answer.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": "BERT and RoBERTa."
  "evidence": ["We compare our approach against two baselines. In Table 1, we compare against BERT. In Table 2, we compare against RoBERTa. Our findings indicate that our approach improves over both baeslines..."]
}

The "answer" field should be roughly 3 characters in length.

Do not include any text in your response other than the json. If the question is unanswerable given the provided excerpts, respond with the single word "null".

To repeat, the question is: How many annotators tagged each tweet?

Answer JSON object:
{"answer": "One", "evidence": ["We developed the annotation guidelines jointly with an experienced annotator, who is a native Arabic speaker with a good knowledge of various Arabic dialects. We made sure that our guidelines were compatible with those of OffensEval2019. The annotator carried out all annotation. Tweets were given one or more of the following four labels: offensive, vulgar, hate speech, or clean. Since the offensive label covers both vulgar and hate speech and vulgarity and hate speech are not mutually exclusive, a tweet can be just offensive or offensive and vulgar and/or hate speech. The annotation adhered to the following guidelines:"]}