You will be shown sections from a scientific research paper, together with a question about the paper. Your task is to answer the question based on the contents of the paper. Your response should be a `json` object with two fields:

"answer": An array of strings extracted directly from the paper which, collectively, answer the question.
"evidence": An array of strings. Each should be an excerpt from the paper, in which one or more of the extracted answers can be found.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": ["BERT", "RoBERTa"],
  "evidence": ["In our experiments, we compare the performance of our model against BERT and RoBERTa."]
}

Do not include any text in your response other than the json.

If the question is unanswerable given the provided excerpts, respond with the single word "null".

Paper: Gender Representation in French Broadcast Corpora and Its Impact on ASR Performance

This paper analyzes the gender representation in four major corpora of French broadcast. These corpora being widely used within the speech processing community, they are a primary material for training automatic speech recognition (ASR) systems. As gender bias has been highlighted in numerous natural language processing (NLP) applications, we study the impact of the gender imbalance in TV and radio broadcast on the performance of an ASR system. This analysis shows that women are under-represented in our data in terms of speakers and speech turns. We introduce the notion of speaker role to refine our analysis and find that women are even fewer within the Anchor category corresponding to prominent speakers. The disparity of available data for both gender causes performance to decrease on women. However this global trend can be counterbalanced for speaker who are used to speak in the media when sufficient amount of data is available.

Methodology ::: Data presentation
Our data consists of two sets used to train and evaluate our automatic speech recognition system. Four major evaluation campaigns have enabled the creation of wide corpora of French broadcast speech: ESTER1 BIBREF13, ESTER2 BIBREF14, ETAPE BIBREF15 and REPERE BIBREF16. These four collections contain radio and/or TV broadcasts aired between 1998 and 2013 which are used by most academic researchers in ASR. Show duration varies between 10min and an hour. As years went by and speech processing research was progressing, the difficulty of the tasks augmented and the content of these evaluation corpora changed. ESTER1 and ESTER2 mainly contain prepared speech such as broadcast news, whereas ETAPE and REPERE consists also of debates and entertainment shows, spontaneous speech introducing more difficulty in its recognition.
Our training set contains 27,085 speech utterances produced by 2,506 speakers, accounting for approximately 100 hours of speech. Our evaluation set contains 74,064 speech utterances produced by 1,268 speakers for a total of 70 hours of speech. Training data by show, medium and speech type is summarized in Table and evaluation data in Table . Evaluation data has a higher variety of shows with both prepared (P) and spontaneous (S) speech type (accented speech from African radio broadcast is also included in the evaluation set).

Methodology ::: Gender bias evaluation procedure of an ASR system performance ::: ASR system
The ASR system used in this work is described in BIBREF25. It uses the KALDI toolkit BIBREF26, following a standard Kaldi recipe. The acoustic model is based on a hybrid HMM-DNN architecture and trained on the data summarized in Table . Acoustic training data correspond to 100h of non-spontaneous speech type (mostly broadcast news) coming from both radio and TV shows. A 5-gram language model is trained from several French corpora (3,323M words in total) using SRILM toolkit BIBREF27. The pronunciation model is developed using the lexical resource BDLEX BIBREF28 as well as automatic grapheme-to-phoneme (G2P) transcription to find pronunciation variants of our vocabulary (limited to 80K). It is important to re-specify here, for further analysis, that our Kaldi pipeline follows speaker adaptive training (SAT) where we train and decode using speaker adapted features (fMLLR-adapted features) in per-speaker mode. It is well known that speaker adaptation acts as an effective procedure to reduce mismatch between training and evaluation conditions BIBREF29, BIBREF26.

Methodology ::: Gender bias evaluation procedure of an ASR system performance ::: Evaluation
Word Error Rate (WER) is a common metric to evaluate ASR performance. It is measured as the sum of errors (insertions, deletions and substitutions) divided by the total number of words in the reference transcription. As we are investigating the impact on performance of speaker's gender and role, we computed the WER for each speaker at the episode (show occurrence) level. Analyzing at such granularity allows us to avoid large WER variation that could be observed at utterance level (especially for short speech turns) but also makes possible to get several WER values for a given speaker, one for each occurrence of a show in which he/she appears on. Speaker's gender was provided by the meta-data and role was obtained using the criteria from Section SECREF6 computed for each show. This enables us to analyze our results across gender and role categories which was done using Wilcoxon rank sum tests also called Mann-Whitney U test (with $\alpha $= 0.001) BIBREF30. The choice of a Wilcoxon rank sum test and not the commonly used t-test is motivated by the non-normality of our data.

Results ::: Descriptive analysis of gender representation in training data ::: Speaker's role representation
Table presents roles' representation in training data and shows that despite the small number of Anchor speakers in our data (3.79%), they nevertheless concentrate 35.71 % of the total speech time.

Results ::: Performance (WER) analysis on evaluation data ::: Impact of gender on WER
As explained in Section SECREF13, WER is the sum of errors divided by the number of words in the transcription reference. The higher the WER, the poorer the system performance. Our 70h evaluation data contains a large amount of spontaneous speech and is very challenging for the ASR system trained on prepared speech: we observe an overall average WER of 42.9% for women and 34.3% for men. This difference of WER between men and women is statistically significant (med(M) = 25%; med(F) = 29%; U = 709040; p-value < 0.001).
However, when observing gender differences across shows, no clear trend can be identified, as shown in Figure FIGREF21. For shows like Africa1 Infos or La Place du Village, we find an average WER lower for women than for men, while the trend is reversed for shows such as Un Temps de Pauchon or Le Masque et la Plume. The disparity of the results depending on the show leads us to believe that other factors may be entangled within the observed phenomenon.

Results ::: Performance (WER) analysis on evaluation data ::: Role and gender interaction
Figure FIGREF25 presents the WER distribution (WER being obtained for each speaker in a show occurrence) according to the speaker's role and gender. It is worth noticing that the gender difference is only significant within the Punctual speakers group. The average WER is of 49.04% for the women and 38.56% for the men with a p-value smaller than $10^{-6}$ (med(F) = 39%; med(M) = 29%; U = 251,450; p-value < 0.001), whereas it is just a trend between male and female Anchors (med(F) = 21%; med(M) = 21%; U = 116,230; p-value = 0.173). This could be explained by the quantity of data available per speaker.

Results ::: Performance (WER) analysis on evaluation data ::: Speech type as a third entangled factor?
In order to try to explain the observed variation in our results depending on shows and gender (Figure FIGREF21), we add the notion of speech type to shed some light on our results. BIBREF22 and BIBREF24 suggested that the speaker professionalism, associated with clear and hyper-articulated speech could be an explaining factor for better performance.
Based on our categorization in prepared speech (mostly news reports) and spontaneous speech (mostly debates and entertainment shows), we cross this parameter in our performance analysis. As shown on Figure FIGREF26, these results confirm the inherent challenge of spontaneous speech compared to prepared speech. WER scores are similar between men and women when considering prepared speech (med(F) = 18%; med(M) = 21%; U = 217,160; p-value = 0.005) whereas they are worse for women (61.29%) than for men (46.51%) with p-value smaller than $10^{-14}$ for the spontaneous speech type (med(F) = 61%; med(M) = 37%; U = 153,580; p-value < 0.001).

Discussion
We find a clear disparity in terms of women presence and speech quantity in French media. Our data being recorded between 1998 and 2013, we can expect this disparity to be smaller on more recent broadcast recordings, especially since the French government displays efforts toward parity in media representation. One can also argue that even if our analysis was conducted on a large amount of data it does not reach the exhaustiveness of large-scale studies such as the one of BIBREF2. Nonetheless it does not affect the relevance of our findings, because if real-world gender representation might be more balanced today, these corpora are still used as training data for AI systems.
The performance difference across gender we observed corroborates (on a larger quantity and variety of language data produced by more than 2400 speakers) the results obtained by BIBREF24 on isolated words recognition. However the following study on read speech does not replicate these results. Yet a performance degradation is observed across dialect and race BIBREF8. BIBREF22 found lower WER for women than men on broadcast news and conversational telephone speech for both English and French. The authors suggest that gender stereotypes associated with women role in education and language acquisition induce a more normative elocution. We observed that the higher the degree of normativity of speech the smaller the gender difference. No significant gender bias is observed for prepared speech nor within the Anchor category. Even if we do not find similar results with lower WER for women than men, we obtained a median WER smaller for women on prepared speech and equal to the male median WER for the Anchor speakers.
Another explanation could be the use of adaptation within the pipeline. Most broadcast programs transcription systems have a speaker adaptation step within their decoding pipeline, which is the case for our system. An Anchor speaker intervening more often would have a larger quantity of data to realize such adaptation of the acoustic model. On the contrary, Punctual speakers who appear scarcely in the data are not provided with the same amount of adaptation data. Hence we can hypothesize that gender performance difference observed for Punctual speakers is due to the fact that female speech is further from the (initial non-adapted) acoustic model as it was trained on unbalanced data (as shown in Table ). Considering that Punctual speakers represent 92.78% of the speakers, this explains why gender difference is significant over our entire data set. A way to confirm our hypothesis would be to reproduce our analysis on WER values obtained without using speaker adapted features at the decoding step.
When decoding prepared speech (hence similar to the training data), no significant difference is found in WER between men and women, revealing that the speaker adaptation step could be sufficient to reach same performance for both genders. But when decoding more spontaneous speech, there is a mismatch with the initial acoustic model (trained on prepared speech). Consequently, the speaker adaptation step might not be enough to recover good ASR performance, especially for women for whom less adaptation data is available (see Section 4.2.3).

Conclusion
This paper has investigated gender bias in ASR performance through the following research questions: i) what is the proportion of men and women in French radio and TV media data ? ii) what is the impact of the observed disparity on ASR performance ? iii) is this as simple as a problem of gender proportion in the training data or are other factors entangled ? Our contributions are the following:
Descriptive analysis of the broadcast data used to train our ASR system confirms the already known disparity, where 65% of the speakers are men, speaking more than 75% of the time.
When investigating WER scores according to gender, speaker's role and speech type, huge variations are observed. We conclude that gender is clearly a factor of variation in ASR performance, with a WER increase of 24% for women compared to men, exhibiting a clear gender bias.
Gender bias varies across speaker's role and speech spontaneity level. Performance for Punctual speakers respectively spontaneous speech seems to reinforce this gender bias with a WER increase of 27.2% respectively 31.8% between male and female speakers.
We found that an ASR system trained on unbalanced data regarding gender produces gender bias performance. Therefore, in order to create fair systems it is necessary to take into account the representation problems in society that are going to be encapsulated in the data. Understanding how women under-representation in broadcast data can lead to bias in ASR performances is the key to prevent re-implementing and reinforcing discrimination already existing in our societies. This is in line with the concept of “Fairness by Design" proposed by BIBREF31.
Gender, race, religion, nationality are all characteristics that we deem unfair to classify on, and these ethical standpoints needs to be taken into account in systems' design. Characteristics that are not considered as relevant in a given task can be encapsulated in data nonetheless, and lead to bias performance. Being aware of the demographic skews our data set might contain is a first step to track the life cycle of a training data set and a necessary step to control the tools we develop.

Table 2: Evaluation data description

Table 5: Train data (percentages are calculated within role’s categories)

Table 3: Gender representation in training data

Figure 3: WER distribution by type of speech and gender - 70h evaluation set

Question: What is the goal of investigating NLP gender bias specifically in the news broadcast domain and Anchor role?
{"answer": ["create fair systems"], "evidence": ["We found that an ASR system trained on unbalanced data regarding gender produces gender bias performance. Therefore, in order to create fair systems it is necessary to take into account the representation problems in society that are going to be encapsulated in the data. Understanding how women under-representation in broadcast data can lead to bias in ASR performances is the key to prevent re-implementing and reinforcing discrimination already existing in our societies. This is in line with the concept of \u201cFairness by Design\" proposed by BIBREF31."]}