You will be shown sections from a scientific research paper, together with a question about the paper. Your task is to answer the question based on the contents of the paper. Your response should be a `json` object with two fields:

"answer": An array of strings extracted directly from the paper which, collectively, answer the question.
"evidence": An array of strings. Each should be an excerpt from the paper, in which one or more of the extracted answers can be found.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": ["BERT", "RoBERTa"],
  "evidence": ["In our experiments, we compare the performance of our model against BERT and RoBERTa."]
}

Do not include any text in your response other than the json.

If the question is unanswerable given the provided excerpts, respond with the single word "null".

Paper: Gender Representation in French Broadcast Corpora and Its Impact on ASR Performance

This paper analyzes the gender representation in four major corpora of French broadcast. These corpora being widely used within the speech processing community, they are a primary material for training automatic speech recognition (ASR) systems. As gender bias has been highlighted in numerous natural language processing (NLP) applications, we study the impact of the gender imbalance in TV and radio broadcast on the performance of an ASR system. This analysis shows that women are under-represented in our data in terms of speakers and speech turns. We introduce the notion of speaker role to refine our analysis and find that women are even fewer within the Anchor category corresponding to prominent speakers. The disparity of available data for both gender causes performance to decrease on women. However this global trend can be counterbalanced for speaker who are used to speak in the media when sufficient amount of data is available.

Introduction
In recent years, gender has become a hot topic within the political, societal and research spheres. Numerous studies have been conducted in order to evaluate the presence of women in media, often revealing their under-representation, such as the Global Media Monitoring Project BIBREF0. In the French context, the CSA BIBREF1 produces a report on gender representation in media on a yearly basis. The 2017 report shows that women represent 40% of French media speakers, with a significant drop during high-audience hours (6:00-8:00pm) reaching a value of only 29%. Another large scale study confirmed this trend with an automatic analysis of gender in French audiovisuals streams, highlighting a huge variation across type of shows BIBREF2.
Besides the social impact of gender representation, broadcast recordings are also a valuable source of data for the speech processing community. Indeed, automatic speech recognition (ASR) systems require large amount of annotated speech data to be efficiently trained, which leaves us facing the emerging concern about the fact that "AI artifacts tend to reflect the goals, knowledge and experience of their creators" BIBREF3. Since we know that women are under-represented in media and that the AI discipline has retained a male-oriented focus BIBREF4, we can legitimately wonder about the impact of using such data as a training set for ASR technologies. This concern is strengthened by the recent works uncovering gender bias in several natural language processing (NLP) tools such as BIBREF5, BIBREF6, BIBREF7, BIBREF8.
In this paper, we first highlight the importance of TV and radio broadcast as a source of data for ASR, and the potential impact it can have. We then perform a statistical analysis of gender representation in a data set composed of four state-of-the-art corpora of French broadcast, widely used within the speech community. Finally we question the impact of such a representation on the systems developed on this data, through the perspective of an ASR system.

Methodology ::: Data presentation
Our data consists of two sets used to train and evaluate our automatic speech recognition system. Four major evaluation campaigns have enabled the creation of wide corpora of French broadcast speech: ESTER1 BIBREF13, ESTER2 BIBREF14, ETAPE BIBREF15 and REPERE BIBREF16. These four collections contain radio and/or TV broadcasts aired between 1998 and 2013 which are used by most academic researchers in ASR. Show duration varies between 10min and an hour. As years went by and speech processing research was progressing, the difficulty of the tasks augmented and the content of these evaluation corpora changed. ESTER1 and ESTER2 mainly contain prepared speech such as broadcast news, whereas ETAPE and REPERE consists also of debates and entertainment shows, spontaneous speech introducing more difficulty in its recognition.
Our training set contains 27,085 speech utterances produced by 2,506 speakers, accounting for approximately 100 hours of speech. Our evaluation set contains 74,064 speech utterances produced by 1,268 speakers for a total of 70 hours of speech. Training data by show, medium and speech type is summarized in Table and evaluation data in Table . Evaluation data has a higher variety of shows with both prepared (P) and spontaneous (S) speech type (accented speech from African radio broadcast is also included in the evaluation set).

Methodology ::: Methodology for descriptive analysis of gender representation in training data
We first describe the gender representation in training data. Gender representation is measured in terms of number of speakers, number of utterances (or speech turns), and turn lengths (descriptive statistics are given in Section SECREF16). Each speech turn was mapped to its speaker in order to associate it with a gender.
As pointed out by the CSA report BIBREF1, women presence tends to be marginal within the high-audience hours, showing that women are represented but less than men and within certain given conditions. It is clear that a small number of speakers is responsible for a large number of speech turns. Most of these speakers are journalists, politicians, presenters and such, who are representative of a show. Therefore, we introduce the notion of speaker's role to refine our exploration of gender disparity, following studies which quantified women's presence in terms of role. Within our work, we define the notion of speaker role by two criteria specifying the speaker's on-air presence, namely the number of speech turns and the cumulative duration of his or her speaking time in a show. Based on the available speech transcriptions and meta-data, we compute for each speaker the number of speech turns uttered as well as their total length. We then use the following criteria to define speaker's role: a speaker is considered as speaking often (respectively seldom) if he/she accumulates a total of turns higher (respectively lower) than 1% of the total number of speech turns in a given show. The same process is applied to identify speakers talking for a long period from those who do not. We end up with two salient roles called Anchors and Punctual speakers:
the Anchor speakers (A) are above the threshold of 1% for both criteria, meaning they are intervening often and for a long time thus holding an important place in interaction;
the Punctual speakers (PS) on the contrary are below the threshold of 1% for both the total number of turns and the total speech time.
These roles are defined at the show level. They could be roughly assimilated to the categorization “host/guest” in radio and TV shows. Anchors could be described as professional speakers, producing mostly prepared speech, whereas Punctual speakers are more likely to be “everyday people". The concept of speaker's role makes sense at both sociological and technical levels. An Anchor speaker is more likely to be known from the audience (society), but he or she will also likely have a professional (clear) way of speaking (as mentioned by BIBREF22 and BIBREF8), as well as a high number of utterances, augmenting the amount of data available for a given gender category.

Results ::: Descriptive analysis of gender representation in training data ::: Gender representation
As expected, we observe a disparity in terms of gender representation in our data (see Table ). Women represent 33.16% of the speakers, confirming the figures given by the GMMP report BIBREF0. However, it is worth noticing that women account for only 22.57% of the total speech time, which leads us to conclude that women also speak less than men.

Results ::: Descriptive analysis of gender representation in training data ::: Speaker's role representation
Table presents roles' representation in training data and shows that despite the small number of Anchor speakers in our data (3.79%), they nevertheless concentrate 35.71 % of the total speech time.

Results ::: Performance (WER) analysis on evaluation data ::: Impact of gender on WER
As explained in Section SECREF13, WER is the sum of errors divided by the number of words in the transcription reference. The higher the WER, the poorer the system performance. Our 70h evaluation data contains a large amount of spontaneous speech and is very challenging for the ASR system trained on prepared speech: we observe an overall average WER of 42.9% for women and 34.3% for men. This difference of WER between men and women is statistically significant (med(M) = 25%; med(F) = 29%; U = 709040; p-value < 0.001).
However, when observing gender differences across shows, no clear trend can be identified, as shown in Figure FIGREF21. For shows like Africa1 Infos or La Place du Village, we find an average WER lower for women than for men, while the trend is reversed for shows such as Un Temps de Pauchon or Le Masque et la Plume. The disparity of the results depending on the show leads us to believe that other factors may be entangled within the observed phenomenon.

Results ::: Performance (WER) analysis on evaluation data ::: Impact of role on WER
Speaker's role seems to have an impact on WER: we obtain an average WER of 30.8% for the Anchor speakers and 42.23% for the Punctual speakers. This difference is statistically significant with a p-value smaller than $10^{-14}$ (med(A) = 21%; med(P) = 31%; U = 540,430; p-value < 0.001) .

Results ::: Performance (WER) analysis on evaluation data ::: Role and gender interaction
Figure FIGREF25 presents the WER distribution (WER being obtained for each speaker in a show occurrence) according to the speaker's role and gender. It is worth noticing that the gender difference is only significant within the Punctual speakers group. The average WER is of 49.04% for the women and 38.56% for the men with a p-value smaller than $10^{-6}$ (med(F) = 39%; med(M) = 29%; U = 251,450; p-value < 0.001), whereas it is just a trend between male and female Anchors (med(F) = 21%; med(M) = 21%; U = 116,230; p-value = 0.173). This could be explained by the quantity of data available per speaker.

Results ::: Performance (WER) analysis on evaluation data ::: Speech type as a third entangled factor?
In order to try to explain the observed variation in our results depending on shows and gender (Figure FIGREF21), we add the notion of speech type to shed some light on our results. BIBREF22 and BIBREF24 suggested that the speaker professionalism, associated with clear and hyper-articulated speech could be an explaining factor for better performance.
Based on our categorization in prepared speech (mostly news reports) and spontaneous speech (mostly debates and entertainment shows), we cross this parameter in our performance analysis. As shown on Figure FIGREF26, these results confirm the inherent challenge of spontaneous speech compared to prepared speech. WER scores are similar between men and women when considering prepared speech (med(F) = 18%; med(M) = 21%; U = 217,160; p-value = 0.005) whereas they are worse for women (61.29%) than for men (46.51%) with p-value smaller than $10^{-14}$ for the spontaneous speech type (med(F) = 61%; med(M) = 37%; U = 153,580; p-value < 0.001).

Conclusion
This paper has investigated gender bias in ASR performance through the following research questions: i) what is the proportion of men and women in French radio and TV media data ? ii) what is the impact of the observed disparity on ASR performance ? iii) is this as simple as a problem of gender proportion in the training data or are other factors entangled ? Our contributions are the following:
Descriptive analysis of the broadcast data used to train our ASR system confirms the already known disparity, where 65% of the speakers are men, speaking more than 75% of the time.
When investigating WER scores according to gender, speaker's role and speech type, huge variations are observed. We conclude that gender is clearly a factor of variation in ASR performance, with a WER increase of 24% for women compared to men, exhibiting a clear gender bias.
Gender bias varies across speaker's role and speech spontaneity level. Performance for Punctual speakers respectively spontaneous speech seems to reinforce this gender bias with a WER increase of 27.2% respectively 31.8% between male and female speakers.
We found that an ASR system trained on unbalanced data regarding gender produces gender bias performance. Therefore, in order to create fair systems it is necessary to take into account the representation problems in society that are going to be encapsulated in the data. Understanding how women under-representation in broadcast data can lead to bias in ASR performances is the key to prevent re-implementing and reinforcing discrimination already existing in our societies. This is in line with the concept of “Fairness by Design" proposed by BIBREF31.
Gender, race, religion, nationality are all characteristics that we deem unfair to classify on, and these ethical standpoints needs to be taken into account in systems' design. Characteristics that are not considered as relevant in a given task can be encapsulated in data nonetheless, and lead to bias performance. Being aware of the demographic skews our data set might contain is a first step to track the life cycle of a training data set and a necessary step to control the tools we develop.

Table 1: Training data description

Table 2: Evaluation data description

Table 5: Train data (percentages are calculated within role’s categories)

Table 3: Gender representation in training data

Table 4: Roles representation in training data

Figure 1: WER scores by show and gender - 70h evaluation set (WER being an error-rate, the smaller the better. If performance were always better for one gender, we would expect each line to have the same slope direction)

Figure 2: WER distribution by role and gender - 70h evaluation set

Figure 3: WER distribution by type of speech and gender - 70h evaluation set

Question: How big is imbalance in analyzed corpora?
{"answer": ["Women represent 33.16% of the speakers"], "evidence": ["As expected, we observe a disparity in terms of gender representation in our data (see Table ). Women represent 33.16% of the speakers, confirming the figures given by the GMMP report BIBREF0. However, it is worth noticing that women account for only 22.57% of the total speech time, which leads us to conclude that women also speak less than men."]}