You will be shown sections from a scientific research paper, together with a question about the paper. Your task is to answer the question based on the contents of the paper. Your response should be a `json` object with two fields:

"answer": An array of strings extracted directly from the paper which, collectively, answer the question.
"evidence": An array of strings. Each should be an excerpt from the paper, in which one or more of the extracted answers can be found.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": ["BERT", "RoBERTa"],
  "evidence": ["In our experiments, we compare the performance of our model against BERT and RoBERTa."]
}

Do not include any text in your response other than the json.

If the question is unanswerable given the provided excerpts, respond with the single word "null".

Paper: Reducing Gender Bias in Word-Level Language Models with a Gender-Equalizing Loss Function

Gender bias exists in natural language datasets which neural language models tend to learn, resulting in biased text generation. In this research, we propose a debiasing approach based on the loss function modification. We introduce a new term to the loss function which attempts to equalize the probabilities of male and female words in the output. Using an array of bias evaluation metrics, we provide empirical evidence that our approach successfully mitigates gender bias in language models without increasing perplexity. In comparison to existing debiasing strategies, data augmentation, and word embedding debiasing, our method performs better in several aspects, especially in reducing gender bias in occupation words. Finally, we introduce a combination of data augmentation and our approach, and show that it outperforms existing strategies in all bias evaluation metrics.

Introduction
Natural Language Processing (NLP) models are shown to capture unwanted biases and stereotypes found in the training data which raise concerns about socioeconomic, ethnic and gender discrimination when these models are deployed for public use BIBREF0 , BIBREF1 .
There are numerous studies that identify algorithmic bias in NLP applications. BIBREF2 showed ethnic bias in Google autocomplete suggestions whereas BIBREF3 found gender bias in advertisement delivery systems. Additionally, BIBREF1 demonstrated that coreference resolution systems exhibit gender bias.
Language modelling is a pivotal task in NLP with important downstream applications such as text generation BIBREF4 . Recent studies by BIBREF0 and BIBREF5 have shown that this task is vulnerable to gender bias in the training corpus. Two prior works focused on reducing bias in language modelling by data preprocessing BIBREF0 and word embedding debiasing BIBREF5 . In this study, we investigate the efficacy of bias reduction during training by introducing a new loss function which encourages the language model to equalize the probabilities of predicting gendered word pairs like he and she. Although we recognize that gender is non-binary, for the purpose of this study, we focus on female and male words.
Our main contributions are summarized as follows: i) to our best knowledge, this study is the first one to investigate bias alleviation in text generation by direct modification of the loss function; ii) our new loss function effectively reduces gender bias in the language models during training by equalizing the probabilities of male and female words in the output; iii) we show that end-to-end debiasing of the language model can achieve word embedding debiasing; iv) we provide an interpretation of our results and draw a comparison to other existing debiasing methods. We show that our method, combined with an existing method, counterfactual data augmentation, achieves the best result and outperforms all existing methods.

Related Work
Recently, the study of bias in NLP applications has received increasing attention from researchers. Most relevant work in this domain can be broadly divided into two categories: word embedding debiasing and data debiasing by preprocessing.

Dataset
For the training data, we use Daily Mail news articles released by BIBREF9 . This dataset is composed of 219,506 articles covering a diverse range of topics including business, sports, travel, etc., and is claimed to be biased and sensational BIBREF5 . For manageability, we randomly subsample 5% of the text. The subsample has around 8.25 million tokens in total.

Language Model
We use a pre-trained 300-dimensional word embedding, GloVe, by BIBREF10 . We apply random search to the hyperparameter tuning of the LSTM language model. The best hyperparameters are as follows: 2 hidden layers each with 300 units, a sequence length of 35, a learning rate of 20 with an annealing schedule of decay starting from 0.25 to 0.95, a dropout rate of 0.25 and a gradient clip of 0.25. We train our models for 150 epochs, use a batch size of 48, and set early stopping with a patience of 5.

Loss Function
Language models are usually trained using cross-entropy loss. Cross-entropy loss at time step INLINEFORM0 is INLINEFORM1 
where INLINEFORM0 is the vocabulary, INLINEFORM1 is the one hot vector of ground truth and INLINEFORM2 indicates the output softmax probability of the model.
We introduce a loss term INLINEFORM0 , which aims to equalize the predicted probabilities of gender pairs such as woman and man. INLINEFORM1 
 INLINEFORM0 and INLINEFORM1 are a set of corresponding gender pairs, INLINEFORM2 is the size of the gender pairs set, and INLINEFORM3 indicates the output softmax probability. We use gender pairs provided by BIBREF7 . By considering only gender pairs we ensure that only gender information is neutralized and distribution over semantic concepts is not altered. For example, it will try to equalize the probabilities of congressman with congresswoman and actor with actress but distribution of congressman, congresswoman versus actor, actress will not be affected. Overall loss can be written as INLINEFORM4 
where INLINEFORM0 is a hyperparameter and INLINEFORM1 is the corpus size. We observe that among the similar minima of the loss function, INLINEFORM2 encourages the model to converge towards a minimum that exhibits the lowest gender bias.

Existing Approaches
We apply CDA where we swap all the gendered words using a bidirectional dictionary of gender pairs described by BIBREF0 . This creates a dataset twice the size of the original data, with exactly the same contextual distributions for both genders and we use it to train the language models.
We also implement the bias regularization method of BIBREF5 which debiases the word embedding during language model training by minimizing the projection of neutral words on the gender axis. We use hyperparameter tuning to find the best regularization coefficient and report results from the model trained with this coefficient. We later refer to this strategy as REG.

Experiments
Initially, we measure the co-occurrence bias in the training data. After training the baseline model, we implement our loss function and tune for the INLINEFORM0 hyperparameter. We test the existing debiasing approaches, CDA and REG, as well but since BIBREF5 reported that results fluctuate substantially with different REG regularization coefficients, we perform hyperparameter tuning and report the best results in Table TABREF12 . Additionally, we implement a combination of our loss function and CDA and tune for INLINEFORM1 . Finally, bias evaluation is performed for all the trained models. Causal occupation bias is measured directly from the models using template datasets discussed above and co-occurrence bias is measured from the model-generated texts, which consist of 10,000 documents of 500 words each.

Results
Results for the experiments are listed in Table TABREF12 . It is interesting to observe that the baseline model amplifies the bias in the training data set as measured by INLINEFORM0 and INLINEFORM1 . From measurements using the described bias metrics, our method effectively mitigates bias in language modelling without a significant increase in perplexity. At INLINEFORM2 value of 1, it reduces INLINEFORM3 by 58.95%, INLINEFORM4 by 45.74%, INLINEFORM5 by 100%, INLINEFORM6 by 98.52% and INLINEFORM7 by 98.98%. Compared to the results of CDA and REG, it achieves the best results in both occupation biases, INLINEFORM8 and INLINEFORM9 , and INLINEFORM10 . We notice that all methods result in INLINEFORM11 around 1, indicating that there are near equal amounts of female and male words in the generated texts. In our experiments we note that with increasing INLINEFORM12 , the bias steadily decreases and perplexity tends to slightly increase. This indicates that there is a trade-off between bias and perplexity.
REG is not very effective in mitigating bias when compared to other methods, and fails to achieve the best result in any of the bias metrics that we used. But REG results in the best perplexity and even does better than the baseline model in this respect. This indicates that REG has a slight regularization effect. Additionally, it is interesting to note that our loss function outperforms REG in INLINEFORM0 even though REG explicitly aims to reduce gender bias in the embeddings. Although our method does not explicitly attempt geometric debiasing of the word embedding, the results show that it results in the most debiased embedding as compared to other methods. Furthermore, BIBREF8 emphasizes that geometric gender bias in word embeddings is not completely understood and existing word embedding debiasing strategies are insufficient. Our approach provides an appealing end-to-end solution for model debiasing without relying on any measure of bias in the word embedding. We believe this concept is generalizable to other NLP applications.
Our method outperforms CDA in INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 . While CDA achieves slightly better results for co-occurrence biases, INLINEFORM3 and INLINEFORM4 , and results in a better perplexity. With a marginal differences, our results are comparable to those of CDA and both models seem to have similar bias mitigation effects. However, our method does not require a data augmentation step and allows training of an unbiased model directly from biased datasets. For this reason, it also requires less time to train than CDA since its training data has a smaller size without data augmentation. Furthermore, CDA fails to effectively mitigate occupation bias when compared to our approach. Although the training data for CDA does not contain gender bias, the model still exhibits some gender bias when measured with our causal occupation bias metrics. This reinforces the concept that some model-level constraints are essential to debiasing a model and dataset debiasing alone cannot be trusted.
Finally, we note that the combination of CDA and our loss function outperforms all the methods in all measures of biases without compromising perplexity. Therefore, it can be argued that a cascade of these approaches can be used to optimally debias the language models.

Conclusion and Discussion
In this research, we propose a new approach for mitigating gender bias in neural language models and empirically show its effectiveness in reducing bias as measured with different evaluation metrics. Our research also highlights the fact that debiasing the model with bias penalties in the loss function is an effective method. We emphasize that loss function based debiasing is powerful and generalizable to other downstream NLP applications. The research also reinforces the idea that geometric debiasing of the word embedding is not a complete solution for debiasing the downstream applications but encourages end-to-end approaches to debiasing.
All the debiasing techniques experimented in this paper rely on a predefined set of gender pairs in some way. CDA used gender pairs for flipping, REG uses it for gender space definition and our technique uses them for computing loss. This reliance on pre-defined set of gender pairs can be considered a limitation of these methods. It also results in another concern. There are gender associated words which do not have pairs, like pregnant. These words are not treated properly by techniques relying on gender pairs.
Future work includes designing a context-aware version of our loss function which can distinguish between the unbiased and biased mentions of the gendered words and only penalize the biased version. Another interesting direction is exploring the application of this method in mitigating racial bias which brings more challenges.

Acknowledgment
We are grateful to Sam Bowman for helpful advice, Shikha Bordia, Cuiying Yang, Gang Qian, Xiyu Miao, Qianyi Fan, Tian Liu, and Stanislav Sobolevsky for discussions, and reviewers for detailed feedback. 

Table 1: Example templates of two types of occupation bias

Table 2: Evaluation results for models trained on Daily Mail and their generated texts

Question: what dataset was used?
{"answer": ["Daily Mail news articles released by BIBREF9 "], "evidence": ["For the training data, we use Daily Mail news articles released by BIBREF9 . This dataset is composed of 219,506 articles covering a diverse range of topics including business, sports, travel, etc., and is claimed to be biased and sensational BIBREF5 . For manageability, we randomly subsample 5% of the text. The subsample has around 8.25 million tokens in total."]}