You will be shown sections from a scientific research paper, together with a question about the paper. Your task is to answer the question based on the contents of the paper. Your response should be a `json` object with two fields:

"answer": An array of strings extracted directly from the paper which, collectively, answer the question.
"evidence": An array of strings. Each should be an excerpt from the paper, in which one or more of the extracted answers can be found.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": ["BERT", "RoBERTa"],
  "evidence": ["In our experiments, we compare the performance of our model against BERT and RoBERTa."]
}

Do not include any text in your response other than the json.

If the question is unanswerable given the provided excerpts, respond with the single word "null".

Paper: Gender Representation in French Broadcast Corpora and Its Impact on ASR Performance

This paper analyzes the gender representation in four major corpora of French broadcast. These corpora being widely used within the speech processing community, they are a primary material for training automatic speech recognition (ASR) systems. As gender bias has been highlighted in numerous natural language processing (NLP) applications, we study the impact of the gender imbalance in TV and radio broadcast on the performance of an ASR system. This analysis shows that women are under-represented in our data in terms of speakers and speech turns. We introduce the notion of speaker role to refine our analysis and find that women are even fewer within the Anchor category corresponding to prominent speakers. The disparity of available data for both gender causes performance to decrease on women. However this global trend can be counterbalanced for speaker who are used to speak in the media when sufficient amount of data is available.

Introduction
In recent years, gender has become a hot topic within the political, societal and research spheres. Numerous studies have been conducted in order to evaluate the presence of women in media, often revealing their under-representation, such as the Global Media Monitoring Project BIBREF0. In the French context, the CSA BIBREF1 produces a report on gender representation in media on a yearly basis. The 2017 report shows that women represent 40% of French media speakers, with a significant drop during high-audience hours (6:00-8:00pm) reaching a value of only 29%. Another large scale study confirmed this trend with an automatic analysis of gender in French audiovisuals streams, highlighting a huge variation across type of shows BIBREF2.
Besides the social impact of gender representation, broadcast recordings are also a valuable source of data for the speech processing community. Indeed, automatic speech recognition (ASR) systems require large amount of annotated speech data to be efficiently trained, which leaves us facing the emerging concern about the fact that "AI artifacts tend to reflect the goals, knowledge and experience of their creators" BIBREF3. Since we know that women are under-represented in media and that the AI discipline has retained a male-oriented focus BIBREF4, we can legitimately wonder about the impact of using such data as a training set for ASR technologies. This concern is strengthened by the recent works uncovering gender bias in several natural language processing (NLP) tools such as BIBREF5, BIBREF6, BIBREF7, BIBREF8.
In this paper, we first highlight the importance of TV and radio broadcast as a source of data for ASR, and the potential impact it can have. We then perform a statistical analysis of gender representation in a data set composed of four state-of-the-art corpora of French broadcast, widely used within the speech community. Finally we question the impact of such a representation on the systems developed on this data, through the perspective of an ASR system.

From gender representation in data to gender bias in AI ::: On the importance of data
The ever growing use of machine learning in science has been enabled by several progresses among which the exponential growth of data available. The quality of a system now depends mostly on the quality and quantity of the data it has been trained on. If it does not discard the importance of an appropriate architecture, it reaffirms the fact that rich and large corpora are a valuable resource. Corpora are research contributions which do not only allow to save and observe certain phenomena or validate a hypothesis or model, but are also a mandatory part of the technology development. This trend is notably observable within the NLP field, where industrial technologies, such as Apple, Amazon or Google vocal assistants now reach high performance level partly due to the amount of data possessed by these companies BIBREF9.
Surprisingly, as data is said to be “the new oil", few data sets are available for ASR systems. The best known are corpora like TIMIT BIBREF10, Switchboard BIBREF11 or Fisher BIBREF12 which date back to the early 1990s. The scarceness of available corpora is justified by the fact that gathering and annotating audio data is costly both in terms of money and time. Telephone conversations and broadcast recordings have been the primary source of spontaneous speech used. Out of all the 130 audio resources proposed by LDC to train automatic speech recognition systems in English, approximately 14% of them are based on broadcast news and conversation. For French speech technologies, four corpora containing radio and TV broadcast are the most widely used: ESTER1 BIBREF13, ESTER2 BIBREF14, ETAPE BIBREF15 and REPERE BIBREF16. These four corpora have been built alongside evaluation campaigns and are still, to our knowledge, the largest French ones of their type available to date.

Methodology ::: Data presentation
Our data consists of two sets used to train and evaluate our automatic speech recognition system. Four major evaluation campaigns have enabled the creation of wide corpora of French broadcast speech: ESTER1 BIBREF13, ESTER2 BIBREF14, ETAPE BIBREF15 and REPERE BIBREF16. These four collections contain radio and/or TV broadcasts aired between 1998 and 2013 which are used by most academic researchers in ASR. Show duration varies between 10min and an hour. As years went by and speech processing research was progressing, the difficulty of the tasks augmented and the content of these evaluation corpora changed. ESTER1 and ESTER2 mainly contain prepared speech such as broadcast news, whereas ETAPE and REPERE consists also of debates and entertainment shows, spontaneous speech introducing more difficulty in its recognition.
Our training set contains 27,085 speech utterances produced by 2,506 speakers, accounting for approximately 100 hours of speech. Our evaluation set contains 74,064 speech utterances produced by 1,268 speakers for a total of 70 hours of speech. Training data by show, medium and speech type is summarized in Table and evaluation data in Table . Evaluation data has a higher variety of shows with both prepared (P) and spontaneous (S) speech type (accented speech from African radio broadcast is also included in the evaluation set).

Methodology ::: Gender bias evaluation procedure of an ASR system performance ::: ASR system
The ASR system used in this work is described in BIBREF25. It uses the KALDI toolkit BIBREF26, following a standard Kaldi recipe. The acoustic model is based on a hybrid HMM-DNN architecture and trained on the data summarized in Table . Acoustic training data correspond to 100h of non-spontaneous speech type (mostly broadcast news) coming from both radio and TV shows. A 5-gram language model is trained from several French corpora (3,323M words in total) using SRILM toolkit BIBREF27. The pronunciation model is developed using the lexical resource BDLEX BIBREF28 as well as automatic grapheme-to-phoneme (G2P) transcription to find pronunciation variants of our vocabulary (limited to 80K). It is important to re-specify here, for further analysis, that our Kaldi pipeline follows speaker adaptive training (SAT) where we train and decode using speaker adapted features (fMLLR-adapted features) in per-speaker mode. It is well known that speaker adaptation acts as an effective procedure to reduce mismatch between training and evaluation conditions BIBREF29, BIBREF26.

Results ::: Descriptive analysis of gender representation in training data ::: Gender representation
As expected, we observe a disparity in terms of gender representation in our data (see Table ). Women represent 33.16% of the speakers, confirming the figures given by the GMMP report BIBREF0. However, it is worth noticing that women account for only 22.57% of the total speech time, which leads us to conclude that women also speak less than men.

Results ::: Descriptive analysis of gender representation in training data ::: Speaker's role representation
Table presents roles' representation in training data and shows that despite the small number of Anchor speakers in our data (3.79%), they nevertheless concentrate 35.71 % of the total speech time.

Results ::: Performance (WER) analysis on evaluation data ::: Impact of gender on WER
As explained in Section SECREF13, WER is the sum of errors divided by the number of words in the transcription reference. The higher the WER, the poorer the system performance. Our 70h evaluation data contains a large amount of spontaneous speech and is very challenging for the ASR system trained on prepared speech: we observe an overall average WER of 42.9% for women and 34.3% for men. This difference of WER between men and women is statistically significant (med(M) = 25%; med(F) = 29%; U = 709040; p-value < 0.001).
However, when observing gender differences across shows, no clear trend can be identified, as shown in Figure FIGREF21. For shows like Africa1 Infos or La Place du Village, we find an average WER lower for women than for men, while the trend is reversed for shows such as Un Temps de Pauchon or Le Masque et la Plume. The disparity of the results depending on the show leads us to believe that other factors may be entangled within the observed phenomenon.

Results ::: Performance (WER) analysis on evaluation data ::: Impact of role on WER
Speaker's role seems to have an impact on WER: we obtain an average WER of 30.8% for the Anchor speakers and 42.23% for the Punctual speakers. This difference is statistically significant with a p-value smaller than $10^{-14}$ (med(A) = 21%; med(P) = 31%; U = 540,430; p-value < 0.001) .

Results ::: Performance (WER) analysis on evaluation data ::: Role and gender interaction
Figure FIGREF25 presents the WER distribution (WER being obtained for each speaker in a show occurrence) according to the speaker's role and gender. It is worth noticing that the gender difference is only significant within the Punctual speakers group. The average WER is of 49.04% for the women and 38.56% for the men with a p-value smaller than $10^{-6}$ (med(F) = 39%; med(M) = 29%; U = 251,450; p-value < 0.001), whereas it is just a trend between male and female Anchors (med(F) = 21%; med(M) = 21%; U = 116,230; p-value = 0.173). This could be explained by the quantity of data available per speaker.

Results ::: Performance (WER) analysis on evaluation data ::: Speech type as a third entangled factor?
In order to try to explain the observed variation in our results depending on shows and gender (Figure FIGREF21), we add the notion of speech type to shed some light on our results. BIBREF22 and BIBREF24 suggested that the speaker professionalism, associated with clear and hyper-articulated speech could be an explaining factor for better performance.
Based on our categorization in prepared speech (mostly news reports) and spontaneous speech (mostly debates and entertainment shows), we cross this parameter in our performance analysis. As shown on Figure FIGREF26, these results confirm the inherent challenge of spontaneous speech compared to prepared speech. WER scores are similar between men and women when considering prepared speech (med(F) = 18%; med(M) = 21%; U = 217,160; p-value = 0.005) whereas they are worse for women (61.29%) than for men (46.51%) with p-value smaller than $10^{-14}$ for the spontaneous speech type (med(F) = 61%; med(M) = 37%; U = 153,580; p-value < 0.001).

Discussion
We find a clear disparity in terms of women presence and speech quantity in French media. Our data being recorded between 1998 and 2013, we can expect this disparity to be smaller on more recent broadcast recordings, especially since the French government displays efforts toward parity in media representation. One can also argue that even if our analysis was conducted on a large amount of data it does not reach the exhaustiveness of large-scale studies such as the one of BIBREF2. Nonetheless it does not affect the relevance of our findings, because if real-world gender representation might be more balanced today, these corpora are still used as training data for AI systems.
The performance difference across gender we observed corroborates (on a larger quantity and variety of language data produced by more than 2400 speakers) the results obtained by BIBREF24 on isolated words recognition. However the following study on read speech does not replicate these results. Yet a performance degradation is observed across dialect and race BIBREF8. BIBREF22 found lower WER for women than men on broadcast news and conversational telephone speech for both English and French. The authors suggest that gender stereotypes associated with women role in education and language acquisition induce a more normative elocution. We observed that the higher the degree of normativity of speech the smaller the gender difference. No significant gender bias is observed for prepared speech nor within the Anchor category. Even if we do not find similar results with lower WER for women than men, we obtained a median WER smaller for women on prepared speech and equal to the male median WER for the Anchor speakers.
Another explanation could be the use of adaptation within the pipeline. Most broadcast programs transcription systems have a speaker adaptation step within their decoding pipeline, which is the case for our system. An Anchor speaker intervening more often would have a larger quantity of data to realize such adaptation of the acoustic model. On the contrary, Punctual speakers who appear scarcely in the data are not provided with the same amount of adaptation data. Hence we can hypothesize that gender performance difference observed for Punctual speakers is due to the fact that female speech is further from the (initial non-adapted) acoustic model as it was trained on unbalanced data (as shown in Table ). Considering that Punctual speakers represent 92.78% of the speakers, this explains why gender difference is significant over our entire data set. A way to confirm our hypothesis would be to reproduce our analysis on WER values obtained without using speaker adapted features at the decoding step.
When decoding prepared speech (hence similar to the training data), no significant difference is found in WER between men and women, revealing that the speaker adaptation step could be sufficient to reach same performance for both genders. But when decoding more spontaneous speech, there is a mismatch with the initial acoustic model (trained on prepared speech). Consequently, the speaker adaptation step might not be enough to recover good ASR performance, especially for women for whom less adaptation data is available (see Section 4.2.3).

Table 1: Training data description

Table 2: Evaluation data description

Table 3: Gender representation in training data

Table 4: Roles representation in training data

Question: What tasks did they use to evaluate performance for male and female speakers?
{"answer": ["ASR"], "evidence": ["In this paper, we first highlight the importance of TV and radio broadcast as a source of data for ASR, and the potential impact it can have. We then perform a statistical analysis of gender representation in a data set composed of four state-of-the-art corpora of French broadcast, widely used within the speech community. Finally we question the impact of such a representation on the systems developed on this data, through the perspective of an ASR system."]}