You will be shown sections from a scientific research paper, together with a question about the paper. Paragraphs in the paper are separated by newlines. Your task is to answer the question based on the contents of the paper.

Paper:
----------------------------------------
Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems

Automatic machine learning systems can inadvertently accentuate and perpetuate inappropriate human biases. Past work on examining inappropriate biases has largely focused on just individual systems. Further, there is no benchmark dataset for examining inappropriate biases in systems. Here for the first time, we present the Equity Evaluation Corpus (EEC), which consists of 8,640 English sentences carefully chosen to tease out biases towards certain races and genders. We use the dataset to examine 219 automatic sentiment analysis systems that took part in a recent shared task, SemEval-2018 Task 1 'Affect in Tweets'. We find that several of the systems show statistically significant bias; that is, they consistently provide slightly higher sentiment intensity predictions for one race or one gender. We make the EEC freely available.

Gender Bias Results
Individual submission results were communicated to the participants. Here, we present the summary results across all the teams. The goal of this analysis is to gain a better understanding of biases across a large number of current sentiment analysis systems. Thus, we partition the submissions into three groups according to the bias they show:
F=M not significant: submissions that showed no statistically significant difference in intensity scores predicted for corresponding female and male noun phrase sentences,
F INLINEFORM0 –M INLINEFORM1 significant: submissions that consistently gave higher scores for sentences with female noun phrases than for corresponding sentences with male noun phrases,
F INLINEFORM0 –M INLINEFORM1 significant: submissions that consistently gave lower scores for sentences with female noun phrases than for corresponding sentences with male noun phrases.
For each system and each sentence pair, we calculate the score difference INLINEFORM0 as the score for the female noun phrase sentence minus the score for the corresponding male noun phrase sentence. Table TABREF24 presents the summary results for each of the bias groups. It has the following columns:
#Subm.: number of submissions in each group.
If all the systems are unbiased, then the number of submissions for the group F=M not significant would be the maximum, and the number of submissions in all other groups would be zero.
Avg. score difference F INLINEFORM0 –M INLINEFORM1 : the average INLINEFORM2 for only those pairs where the score for the female noun phrase sentence is higher. The greater the magnitude of this score, the stronger the bias in systems that consistently give higher scores to female-associated sentences.
Avg. score difference F INLINEFORM0 –M INLINEFORM1 : the average INLINEFORM2 for only those pairs where the score for the female noun phrase sentence is lower. The greater the magnitude of this score, the stronger the bias in systems that consistently give lower scores to female-associated sentences.
Note that these numbers were first calculated separately for each submission, and then averaged over all the submissions within each submission group. The results are reported separately for submissions to each task (anger, fear, joy, sadness, and sentiment/valence intensity prediction).
Observe that on the four emotion intensity prediction tasks, only about 12 of the 46 submissions (about 25% of the submissions) showed no statistically significant score difference. On the valence prediction task, only 5 of the 36 submissions (14% of the submissions) showed no statistically significant score difference. Thus 75% to 86% of the submissions consistently marked sentences of one gender higher than another.
When predicting anger, joy, or valence, the number of systems consistently giving higher scores to sentences with female noun phrases (21–25) is markedly higher than the number of systems giving higher scores to sentences with male noun phrases (8–13). (Recall that higher valence means more positive sentiment.) In contrast, on the fear task, most submissions tended to assign higher scores to sentences with male noun phrases (23) as compared to the number of systems giving higher scores to sentences with female noun phrases (12). When predicting sadness, the number of submissions that mostly assigned higher scores to sentences with female noun phrases (18) is close to the number of submissions that mostly assigned higher scores to sentences with male noun phrases (16). These results are in line with some common stereotypes, such as females are more emotional, and situations involving male agents are more fearful BIBREF27 .
Figure FIGREF25 shows the score differences ( INLINEFORM0 ) for individual systems on the valence regression task. Plots for the four emotion intensity prediction tasks are shown in Figure FIGREF31 in the Appendix. Each point ( ▲, ▼, ●) on the plot corresponds to the difference in scores predicted by the system on one sentence pair. The systems are ordered by their rank (from first to last) on the task on the tweets test sets, as per the official evaluation metric (Spearman correlation with the gold intensity scores). We will refer to the difference between the maximal value of INLINEFORM1 and the minimal value of INLINEFORM2 for a particular system as the INLINEFORM3 –spread. Observe that the INLINEFORM4 –spreads for many systems are rather large, up to 0.57. Depending on the task, the top 10 or top 15 systems as well as some of the worst performing systems tend to have smaller INLINEFORM5 –spreads while the systems with medium to low performance show greater sensitivity to the gender-associated words. Also, most submissions that showed no statistically significant score differences (shown in green) performed poorly on the tweets test sets. Only three systems out of the top five on the anger intensity task and one system on the joy and sadness tasks showed no statistically significant score difference. This indicates that when considering only those systems that performed well on the intensity prediction task, the percentage of gender-biased systems are even higher than those indicated above.
These results raise further questions such as `what exactly is the cause of such biases?' and `why is the bias impacted by the emotion task under consideration?'. Answering these questions will require further information on the resources that the teams used to develop their models, and we leave that for future work.
Average score differences: For submissions that showed statistically significant score differences, the average score difference F INLINEFORM0 –M INLINEFORM1 and the average score difference F INLINEFORM2 –M INLINEFORM3 were INLINEFORM4 . Since the intensity scores range from 0 to 1, 0.03 is 3% of the full range. The maximal score difference ( INLINEFORM5 ) across all the submissions was as high as 0.34. Note, however, that these INLINEFORM6 s are the result of changing just one word in a sentence. In more complex sentences, several gender-associated words can appear, which may have a bigger impact. Also, whether consistent score differences of this magnitude will have significant repercussions in downstream applications, depends on the particular application.
Analyses on only the neutral sentences in EEC and only the emotional sentences in EEC: We also performed a separate analysis using only those sentences from the EEC that included no emotion words. Recall that there are four templates that contain no emotion words. Tables TABREF26 shows these results. We observe similar trends as in the analysis on the full set. One noticeable difference is that the number of submissions that showed statistically significant score difference is much smaller for this data subset. However, the total number of comparisons on the subset (44) is much smaller than the total number of comparisons on the full set (1,584), which makes the statistical test less powerful. Note also that the average score differences on the subset (columns 3 and 4 in Table TABREF26 ) tend to be higher than the differences on the full set (columns 3 and 4 in Table TABREF24 ). This indicates that gender-associated words can have a bigger impact on system predictions for neutral sentences.
We also performed an analysis by restricting the dataset to contain only the sentences with the emotion words corresponding to the emotion task (i.e., submissions to the anger intensity prediction task were evaluated only on sentences with anger words). The results (not shown here) were similar to the results on the full set.

Race Bias Results
We did a similar analysis for race as we did for gender. For each submission on each task, we calculated the difference between the average predicted score on the set of sentences with African American (AA) names and the average predicted score on the set of sentences with European American (EA) names. Then, we aggregated the results over all such sentence pairs in the EEC.
Table TABREF29 shows the results. The table has the same form and structure as the gender result tables. Observe that the number of submissions with no statistically significant score difference for sentences pertaining to the two races is about 5–11 (about 11% to 24%) for the four emotions and 3 (about 8%) for valence. These numbers are even lower than what was found for gender.
The majority of the systems assigned higher scores to sentences with African American names on the tasks of anger, fear, and sadness intensity prediction. On the joy and valence tasks, most submissions tended to assign higher scores to sentences with European American names. These tendencies reflect some common stereotypes that associate African Americans with more negative emotions BIBREF28 .
Figure FIGREF28 shows the score differences for individual systems on race sentence pairs on the valence regression task. Plots for the four emotion intensity prediction tasks are shown in Figure FIGREF32 in the Appendix. Here, the INLINEFORM0 –spreads are smaller than on the gender sentence pairs—from 0 to 0.15. As in the gender analysis, on the valence task the top 13 systems as well as some of the worst performing systems have smaller INLINEFORM1 –spread while the systems with medium to low performance show greater sensitivity to the race-associated names. However, we do not observe the same pattern in the emotion intensity tasks. Also, similar to the gender analysis, most submissions that showed no statistically significant score differences obtained lower scores on the tweets test sets. Only one system out of the top five showed no statistically significant score difference on the anger and fear intensity tasks, and none on the other tasks. Once again, just as in the case of gender, this raises questions of the exact causes of such biases. We hope to explore this in future work.

Conclusions and Future Work
We created the Equity Evaluation Corpus (EEC), which consists of 8,640 sentences specifically chosen to tease out gender and race biases in natural language processing systems. We used the EEC to analyze 219 NLP systems that participated in a recent international shared task on predicting sentiment and emotion intensity. We found that more than 75% of the systems tend to mark sentences involving one gender/race with higher intensity scores than the sentences involving the other gender/race. We found such biases to be more widely prevalent for race than for gender. We also found that the bias can be different depending on the particular affect dimension involved.
We found the score differences across genders and across races to be somewhat small on average ( INLINEFORM0 , which is INLINEFORM1 of the 0 to 1 score range). However, for some systems the score differences reached as high as 0.34 (34%). What impact a consistent bias, even with an average magnitude INLINEFORM2 , might have in downstream applications merits further investigation.
We plan to extend the EEC with sentences associated with country names, professions (e.g., doctors, police officers, janitors, teachers, etc.), fields of study (e.g., arts vs. sciences), as well as races (e.g., Asian, mixed, etc.) and genders (e.g., agender, androgyne, trans, queer, etc.) not included in the current study. We can then use the corpus to examine biases across each of those variables as well. We are also interested in exploring which systems (or what techniques) accentuate inappropriate biases in the data and which systems mitigate such biases. Finally, we are interested in exploring how the quality of sentiment analysis predictions varies when applied to text produced by different demographic groups, such as people of different races, genders, and ethnicities.
The Equity Evaluation Corpus and the proposed methodology to examine bias are not meant to be comprehensive. However, using several approaches and datasets such as the one proposed here can bring about a more thorough examination of inappropriate biases in modern machine learning systems.
----------------------------------------

Question: Which race and gender are given higher sentiment intensity predictions?

To answer the question, format your response as a `json` object with two fields:

"answer": A string providing a succinct answer to the question, in your own words.
"evidence": An array of strings. Each entry should be a full paragraph from the paper. Together, the evidence should serve as a justification for the answer.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": "BERT and RoBERTa."
  "evidence": ["We compare our approach against two baselines. In Table 1, we compare against BERT. In Table 2, we compare against RoBERTa. Our findings indicate that our approach improves over both baeslines..."]
}

The "answer" field should be roughly 340 characters in length.

Do not include any text in your response other than the json. If the question is unanswerable given the provided excerpts, respond with the single word "null".

To repeat, the question is: Which race and gender are given higher sentiment intensity predictions?

Answer JSON object:
{"answer": "Females are given higher sentiment intensity when predicting anger, joy or valence, but males are given higher sentiment intensity when predicting  fear.\nAfrican American names are given higher score on the tasks of anger, fear, and sadness intensity prediction,  but European American names are given higher scores on joy and valence task.", "evidence": ["When predicting anger, joy, or valence, the number of systems consistently giving higher scores to sentences with female noun phrases (21\u201325) is markedly higher than the number of systems giving higher scores to sentences with male noun phrases (8\u201313). (Recall that higher valence means more positive sentiment.) In contrast, on the fear task, most submissions tended to assign higher scores to sentences with male noun phrases (23) as compared to the number of systems giving higher scores to sentences with female noun phrases (12). When predicting sadness, the number of submissions that mostly assigned higher scores to sentences with female noun phrases (18) is close to the number of submissions that mostly assigned higher scores to sentences with male noun phrases (16). These results are in line with some common stereotypes, such as females are more emotional, and situations involving male agents are more fearful BIBREF27 .", "The majority of the systems assigned higher scores to sentences with African American names on the tasks of anger, fear, and sadness intensity prediction. On the joy and valence tasks, most submissions tended to assign higher scores to sentences with European American names. These tendencies reflect some common stereotypes that associate African Americans with more negative emotions BIBREF28 ."]}