You will be shown sections from a scientific research paper, together with a question about the paper. Paragraphs in the paper are separated by newlines. Your task is to answer the question based on the contents of the paper.

Paper:
----------------------------------------
Arabic Diacritic Recovery Using a Feature-Rich biLSTM Model

Diacritics (short vowels) are typically omitted when writing Arabic text, and readers have to reintroduce them to correctly pronounce words. There are two types of Arabic diacritics: the first are core-word diacritics (CW), which specify the lexical selection, and the second are case endings (CE), which typically appear at the end of the word stem and generally specify their syntactic roles. Recovering CEs is relatively harder than recovering core-word diacritics due to inter-word dependencies, which are often distant. In this paper, we use a feature-rich recurrent neural network model that uses a variety of linguistic and surface-level features to recover both core word diacritics and case endings. Our model surpasses all previous state-of-the-art systems with a CW error rate (CWER) of 2.86\% and a CE error rate (CEER) of 3.7% for Modern Standard Arabic (MSA) and CWER of 2.2% and CEER of 2.5% for Classical Arabic (CA). When combining diacritized word cores with case endings, the resultant word error rate is 6.0% and 4.3% for MSA and CA respectively. This highlights the effectiveness of feature engineering for such deep neural models.

Our Diacritizer ::: Training and Test Corpora
For MSA, we acquired the diacritized corpus that was used to train the RDI BIBREF7 diacritizer and the Farasa diacritizer BIBREF31. The corpus contains 9.7M tokens with approximately 194K unique surface forms (excluding numbers and punctuation marks). The corpus covers multiple genres such as politics and sports and is a mix of MSA and CA. This corpus is considerably larger than the Arabic Treebank BIBREF35 and is more consistent in its diacritization. For testing, we used the freely available WikiNews test set BIBREF31, which is composed of 70 MSA WikiNews articles (18,300 tokens) and evenly covers a variety of genres including politics, economics, health, science and technology, sports, arts and culture.
For CA, we obtained a large collection of fully diacritized classical texts (2.7M tokens) from a book publisher, and we held-out a small subset of 5,000 sentences (approximately 400k words) for testing. Then, we used the remaining sentences to train the CA models.

Our Diacritizer ::: Core Word Diacritization ::: Features.
Arabic words are typically derived from a limited set of roots by fitting them into so-called stem-templates (producing stems) and may accept a variety of prefixes and suffixes such as prepositions, determiners, and pronouns (producing words). Word stems specify the lexical selection and are typically unaffected by the attached affixes. We used 4 feature types, namely:
CHAR: the characters.
SEG: the position of the character in a word segment. For example, given the word “wAlktAb” (والكتاب> and the book/writers), which is composed of 3 segments “w+Al+ktAb” (و+ال+كتاب>). Letters were marked as “B” if they begin a segment, “M” if they are in the middle of a segment, “E” if they end a segment, and “S” if they are single letter segments. So for “w+Al+ktAb”, the corresponding character positions are “S+BE+BMME”. We used Farasa to perform segmentation, which has a reported segmentation accuracy of 99% on the WikiNews dataset BIBREF36.
PRIOR: diacritics seen in the training set per segment. Since we used a character level model, this feature informed the model with word level information. For example, the word “ktAb” (كتاب>) was observed to have two diacritized forms in the training set, namely “kitaAb” (كِتَاب> – book) and “kut$\sim $aAb” (كُتَّاب> – writers). The first letter in the word (“k”) accepted the diacritics “i” and “u”. Thus given a binary vector representing whether a character is allowed to assume any of the eight primitive Arabic diacritic marks (a, i, u, o, K, N, F, and $\sim $ in order), the first letter would be given the following vector “01100000”. If a word segment was never observed during training, the vector for all letters therein would be set to 11111111. This feature borrows information from HMM models, which have been fairly successful in diacritizing word cores.
CASE: whether the letter expects a core word diacritic or a case ending. Case endings are placed on only one letter in a word, which may or may not be the last letter in the word. This is a binary feature.

Our Diacritizer ::: Core Word Diacritization ::: DNN Model.
Using a DNN model, particularly with a biLSTM BIBREF37, is advantageous because the model automatically explores the space of feature combinations and is able to capture distant dependencies. A number of studies have explored various biLSTM architectures BIBREF17, BIBREF7, BIBREF32 including stacked biLSTMs confirming their effectiveness. As shown in Figure FIGREF14, we employed a character-based biLSTM model with associated features for each character. Every input character had an associated list of $m$ features, and we trained randomly initialized embeddings of size 50 for each feature. Then, we concatenated the feature embeddings vectors creating an $m\times 50$ vector for each character, which was fed into the biLSTM layer of length 100. The output of the biLSTM layer was fed directly into a dense layer of size 100. We used early stopping with patience of 5 epochs, a learning rate of 0.001, a batch size of 256, and an Adamax optimizer. The input was the character sequence in a sentence with words being separated by word boundary markers (WB), and we set the maximum sentence length to 1,250 characters.

Our Diacritizer ::: Case Ending Diacritization ::: Features.
Table TABREF17 lists the features that we used for CE recovery. We used Farasa to perform segmentation and POS tagging and to determine stem-templates BIBREF31. Farasa has a reported POS accuracy of 96% on the WikiNews dataset BIBREF31. Though the Farasa diacritizer utilizes a combination of some the features presented herein, namely segmentation, POS tagging, and stem templates, Farasa's SVM-ranking approach requires explicit specification of feature combinations (ex. $Prob(CE\Vert current\_word, prev\_word, prev\_CE)$). Manual exploration of the feature space is undesirable, and ideally we would want our learning algorithm to do so automatically. The flexibility of the DNN model allowed us to include many more surface level features such as affixes, leading and trailing characters in words and stems, and the presence of words in large gazetteers of named entities. As we show later, these additional features significantly lowered CEER.

Our Diacritizer ::: Case Ending Diacritization ::: DNN Model
Figure FIGREF19 shows the architecture of our DNN algorithm. Every input word had an associated list of $n$ features, and we trained randomly initialized embeddings of size 100 for each feature. Then, we concatenated the feature embeddings vectors creating an $n\times 100$ vector for each word. We fed these vectors into a biLSTM layer of 100 dimensions after applying a dropout of 75%, where dropout behaves like a regularlizer to avoid overfitting BIBREF40. We conducted side experiments with lower dropout rates, but the higher dropout rate worked best. The output of the biLSTM layer was fed into a 100 dimensional dense layer with 15% dropout and softmax activation. We conducted side experiments where we added additional biLSTM layers and replaced softmax with a conditional random field layer, but we did not observe improvements. Thus, we opted for a simpler model. We used a validation set to determine optimal parameters such as dropout rate. Again, we used the “Adamax” optimizer with categorical cross entropy loss and a learning rate of 0.001. We also applied early stopping with patience of up to 5 consecutive epochs without improvement.

Experiments and Results ::: Core Word ::: Results and Error analysis ::: MSA Results:
For MSA, though the CHAR+PRIOR feature led to worse results than using CHAR alone, the results show that combining all the features achieved the best results. Moreover, post correction improved results overall. We compare our results to five other systems, namely Farasa BIBREF31, MADAMIRA BIBREF29, RDI (Rashwan et al., 2015), MIT (Belinkow and Glass, 2015), and Microsoft ATKS BIBREF28. Table TABREF34 compares our system with others in the aforementioned systems. As the results show, our results beat the current state-of-the-art.
For error analysis, we analyzed all the errors (527 errors). The errors types along with examples of each are shown in Table TABREF29. The most prominent error type arises from the selection of a valid diacritized form that does not match the context (40.8%). Perhaps, including POS tags as a feature or augmenting the PRIOR feature with POS tag information and a bigram language model may reduce the error rate further. The second most common error is due to transliterated foreign words including foreign named entities (23.5%). Such words were not observed during training. Further, Arabic Named entities account for 10.6% of the errors, where they were either not seen in training or they share identical non-diacritized forms with other words. Perhaps, building larger gazetteers of diacritized named entities may resolve NE related errors. In 10.8% of the cases, the diacritizer produced in completely incorrect diacritized forms. In some the cases (9.1%), though the diacritizer produced a form that is different from the reference, both forms were in fact correct. Most of these cases were due to variations in diacritization conventions (ex. “bare alef” (A) at start of a word receiving a diacritic or not). Other cases include foreign words and some words where both diacritized forms are equally valid.

Experiments and Results ::: Case Ending ::: Results and Error Analysis
Table TABREF45 lists the results of our setups compared to other systems.

Experiments and Results ::: Case Ending ::: Results and Error Analysis ::: MSA Results:
As the results show, our baseline DNN system outperforms all state-of-the-art systems. Further, adding more features yielded better results overall. Surface-level features resulted in the most gain, followed by POS tags, and lastly stem templates. Further, adding head and tail characters along with a list of sukun words and named entities led to further improvement. Our proposed feature-rich system has a CEER that is approximately 61% lower than any of the state-of-the-art systems.
Figure FIGREF49 shows CE distribution and prediction accuracy. For the four basic markers kasra, fatHa, damma and sukun, which appear 27%, 14%, 9% and 10% respectively, the system has CEER of $\sim $1% for each. Detecting the virtual CE mark is a fairly easy task. All other CE markers represent 13% with almost negligible errors. Table TABREF30 lists a thorough breakdown of all errors accounting for at 1% of the errors along with the most common reasons of the errors and examples illustrating these reasons. For example, the most common error type involves guessing a fatHa (a) instead of damma (u) or vice versa (19.3%). The most common reasons for this error type, based on inspecting the errors, were due to: POS errors (ex. a word is tagged as a verb instead of a noun); and a noun is treated as a subject instead of an object or vice versa. The table details the rest of the error types. Overall, some of the errors are potentially fixable using better POS tagging, improved detection of non-Arabized foreign names, and detection of indeclinability. However, some errors are more difficult and require greater understanding of semantics such as improper attachment, incorrect idafa, and confusion between subject and object. Perhaps, such semantic errors can be resolved using parsing.

Experiments and Results ::: Case Ending ::: Results and Error Analysis ::: CA Results:
The results show that the POS tagging features led to the most improvements followed by the surface features. Combining all features led to the best results with WER of 2.5%. As we saw for CW diacritics, using our best MSA system to diacritize CA led to significantly lower results with CEER of 8.9%.
Figure FIGREF50 shows CE distribution and prediction accuracy. For the four basic markers fatHa, kasra, damma and sukun, which appear 18%, 14%, 13% and 8% respectively, the system has CEER $\sim $0.5% for each. Again, detecting the virtual CE mark was a fairly easy task. All other CE markers representing 20% have negligible errors.
Table TABREF31 lists all the error types, which account for at least 1% of the errors, along with their most common causes and explanatory examples. The error types are similar to those observed for MSA. Some errors are more syntactic and morphological in nature and can be addressed using better POS tagging and identification of indeclinability, particularly as they relate to named entities and nouns with feminine markers. Other errors such as incorrect attachment, incorrect idafa, false subject, and confusion between subject and object can perhaps benefit from the use of parsing. As with the core-word errors for CA, the reference has some errors (ex. {a,i,o} $\Rightarrow $ #), and extra rounds of reviews of the reference are in order.

Fig. 2. DNN case ending model architecture

Table 6. Error analysis: Core word error types for CA

Table 7. Comparing our system to state-of-the-art systems – Core word diacritics

Table 8. MSA Results and comparison to other systems
----------------------------------------

Question: what linguistics features are used?

To answer the question, format your response as a `json` object with two fields:

"answer": A string providing a succinct answer to the question, in your own words.
"evidence": An array of strings. Each entry should be a full paragraph from the paper. Together, the evidence should serve as a justification for the answer.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": "BERT and RoBERTa."
  "evidence": ["We compare our approach against two baselines. In Table 1, we compare against BERT. In Table 2, we compare against RoBERTa. Our findings indicate that our approach improves over both baeslines..."]
}

The "answer" field should be roughly 31 characters in length.

Do not include any text in your response other than the json. If the question is unanswerable given the provided excerpts, respond with the single word "null".

To repeat, the question is: what linguistics features are used?

Answer JSON object:
{"answer": "POS, gender/number and stem POS", "evidence": ["Table TABREF17 lists the features that we used for CE recovery. We used Farasa to perform segmentation and POS tagging and to determine stem-templates BIBREF31. Farasa has a reported POS accuracy of 96% on the WikiNews dataset BIBREF31. Though the Farasa diacritizer utilizes a combination of some the features presented herein, namely segmentation, POS tagging, and stem templates, Farasa\u0027s SVM-ranking approach requires explicit specification of feature combinations (ex. $Prob(CE\\Vert current\\_word, prev\\_word, prev\\_CE)$). Manual exploration of the feature space is undesirable, and ideally we would want our learning algorithm to do so automatically. The flexibility of the DNN model allowed us to include many more surface level features such as affixes, leading and trailing characters in words and stems, and the presence of words in large gazetteers of named entities. As we show later, these additional features significantly lowered CEER."]}