\section{Fairness Composition in Ensembles}

In this section, we evaluate the fairness of various type of ensemble models, divided into homogeneous and heterogeneous techniques, including bagging, boosting, stacking, etc. To ensure consistencies whilst comparing different metrics in the same table, Disparate Impact (DI) is converted to log scale. A value of 0 for any fairness metric suggests that there is no bias in the model. A positive value of a fairness metric suggests the model is biased against the privileged group whereas a negative value suggests bias against unprivileged group.

\textbf{Finding 1:}
\textbf{Boosting}

\textit{Dropout can improve fairness in larger datasets.} Vinayak and Gilad-Bachrach \cite{rashmi2015dart} proposed DART, a dropout technique derived from deep neural networks, for boosted trees. An ensemble of boosted regression trees are known for delivering high quality performances but they suffer from over-specification i.e. the trees added at the end have little contribution to the final result. DART alleviates this problem by introducing dropouts. This method works by selecting only a random subset of the already existing ensembles to compute the gradient that the next tree will fit. The number of trees dropped corresponds to the amount of regularization. In XGBoost, this rate is controlled by the parameter \textit{rate drop} with the range $[0,1]$. No trees are dropped on the lower end of this ratio, while on the higher extreme, all trees are dropped, which makes it similar to a Random Forest classifier.

We investigate the efficacy of DART in reducing unfairness in boosting ensemble models by comparing its performance with the default XGBoost booster \textit{gbtree}. We analyze the performance of the dropout technique on all boosting models in each dataset and record the average change in each fairness metric. The average change is shown in Figure \ref{Dart:comparison}.

\begin{figure}[!htbp]
  \centering

    \includegraphics[width=\linewidth]{Figures/Dropout_figure.pdf}
    \caption{Change of performance and total bias when using DART (negative value denotes reduction in bias and accuracy performance}
    \label{Dart:comparison}
\end{figure}

Interestingly, our analysis shows that for larger datasets (Adult Census, Bank Marketing) we see an improvement across all fairness metrics except \textbf{Thiel Index}. However, for smaller datasets the exact opposite is seen. Our results suggest that using the dropout technique (DART) in boosting models can improve fairness performance at a comparable accuracy performance, in certain situations. Furthermore, it provides important insight into the affect of dropout on fairness, which can be further tuned to align with fairness goals, instead of accuracy.

\textbf{Finding2 :} \textbf{Randomness of feature split.}

One of the ways bagging models like Random Forest improve variance of the model is to induce randomness to the process of model building e.g. randomly selecting features. It has proven to be an effective way of reducing model bias (and hence fairness bias) when used in tandem with multiple learners and bagging. Building on this, Extra Trees introduces additional randomness by randomly selecting not only the subset of the features, but also randomly finding the splits for each feature and then finally taking the best split among those. Random Forest simply finds the best split for each feature. This begs the question, does randomness at feature splitting level cause Tree-based bagging models to be unfair?

To investigate this, we compare Random Forest and Extra Tree models in our benchmark. To ensure consistency in the rest of the parameter space and data split, we only choose those kernels that implement both Extra Trees and Random Forest. Bank Marketing and German Credit did not have any such kernels so we used the same kernel for both the models. For biased datasets such as Titanic and Adult (less biased), Extra Tree models with random splits were more biased compared to optimal splits (RF) Figure \ref{RF-ET:comparison}. This is especially interesting given that the model accuracy performance for models is similar or slightly lower, as can be expected when the optimal split is not chosen. However, it is important to note that for more fair models (BM and GC), randomness as little to no effect on fairness. In terms of bias mitigation methods, our results suggest randomness in the point of split might not be an effective way to tackle bias in decision-tree based models.

\begin{figure}[!htbp]
  \centering

    \includegraphics[width=\linewidth]{Figures/Randomness of feature split.pdf}
    \caption{Change of performance and total bias when using DART (negative value denotes reduction in bias and accuracy performance}
    \label{RF-ET:comparison}
\end{figure}

\textbf{Finding 3: Soft Vs Hard Voting.}

A voting classifier is an ensemble method where the target label is assigned based on the probabilities each individual model within the classifier, assigns the chosen class variable. Voting classifier are of two types, Soft Voting and Hard Voting. In Hard Voting, the predicted target label is the label with majority of votes from each individual learner. Suppose we have 3 classifiers and they output (1,1,0) for the binary classification, then 1 is chosen as the final prediction due to majority voting. On the other hand, soft voting takes the average of the probabilities that each individual learner gives to each class e.g. (0.63, 0.51, 0.20) and (0.37, 0.49, 0.80) is the probabilities given by the 3 models in the voting classifiers for output class A,B respectively. Class A has an average probability 0.45 and Class B 0.55. Since, class B has a higher probability, we assign class B as the final prediction.

Here, we compare the effect of Soft Voting and Hard Voting on Fairness. We observe that for all fairness metrics except \textit{TI}, we see that soft voting outperforms hard voting in Titanic but vice-versa for the other 3 datasets, as shown in Table \ref{Soft vs Hard}. This difference in behavior can be attributed to Titanic's very biased dataset (SPD: ).
\begin{table}[htbp]
  \centering
  \footnotesize
  \caption{Soft vs Hard Voting}
  \renewcommand{\tabcolsep}{1pt}
\resizebox{\columnwidth}{!}{    \begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|c|c|}
    \hline
    \multirow{8}{*}{\rotatebox[origin=c]{90}{Soft}} & & TM-VC1 & TM-VC2 & TM-VC3 & TM-VC4 & TM-VC5 & TM-VC6 & GC-VC1 & AC-VC1 & AC-VC2 & BM-VC1 & BM-VC2 \\
\cline{2-13}          & Acc & 0.853 & 0.821 & 0.835 & 0.839 & 0.810 & 0.813 & 0.757 & 0.828 & 0.826 & 0.915 & 0.937 \\
\cline{2-13}          & F1 & 0.808 & 0.784 & 0.756 & 0.796 & 0.738 & 0.731 & 0.517 & 0.507 & 0.560 & 0.569 & 0.938 \\
\cline{2-13}          & SPD & -0.764 & -0.798 & -0.737 & -0.778 & -0.772 & -0.585 & -0.093 & -0.095 & -0.140 & 0.174 & 0.253 \\
\cline{2-13}          & EOD & -0.579 & -0.701 & -0.441 & -0.674 & -0.625 & -0.482 & -0.026 & -0.074 & -0.143 & 0.144 & 0.019 \\
\cline{2-13}          & AOD & 0.538 & 0.631 & 0.534 & 0.581 & 0.629 & 0.376 & 0.107 & 0.048 & 0.097 & 0.127 & 0.044 \\
\cline{2-13}          & DI & -2.268 & -2.348 & -2.063 & -2.162 & -2.099 & -1.811 & -0.118 & -1.186 & -1.427 & 1.141 & 0.406 \\
\cline{2-13}          & TI & 0.119 & 0.146 & 0.098 & 0.116 & 0.124 & 0.155 & 0.116 & 0.176 & 0.168 & 0.067 & 0.036 \\ \cline{1-13}
    \multirow{7}{*}{\rotatebox[origin=c]{90}{Hard}} & Acc & 0.853 & 0.813 & 0.858 & 0.835 & 0.799 & 0.813 & 0.750 & 0.823 & 0.813 & 0.917 & 0.936 \\
\cline{2-13}          & F1    & 0.800 & 0.773 & 0.784 & 0.792 & 0.724 & 0.731 & 0.468 & 0.457 & 0.440 & 0.556 & 0.937 \\
\cline{2-13}          & SPD   & \cellcolor[rgb]{ .573,  .816,  .314}-0.746 & \cellcolor[rgb]{ .753,  0,  0}-0.826 & \cellcolor[rgb]{ .753,  0,  0}-0.793 & \cellcolor[rgb]{ .753,  0,  0}-0.788 & \cellcolor[rgb]{ .753,  0,  0}-0.783 & \cellcolor[rgb]{ .573,  .816,  .314}-0.568 & \cellcolor[rgb]{ .573,  .816,  .314}-0.062 & \cellcolor[rgb]{ .573,  .816,  .314}-0.072 & \cellcolor[rgb]{ .573,  .816,  .314}-0.067 & \cellcolor[rgb]{ .573,  .816,  .314}0.137 & \cellcolor[rgb]{ .753,  0,  0}0.262 \\
\cline{2-13}          & EOD   & \cellcolor[rgb]{ .753,  0,  0}-0.618 & \cellcolor[rgb]{ .753,  0,  0}-0.757 & \cellcolor[rgb]{ .753,  0,  0}-0.517 & \cellcolor[rgb]{ .753,  0,  0}-0.674 & \cellcolor[rgb]{ .753,  0,  0}-0.656 & \cellcolor[rgb]{ .573,  .816,  .314}-0.437 & \cellcolor[rgb]{ .753,  0,  0}-0.077 & \cellcolor[rgb]{ .573,  .816,  .314}-0.042 & \cellcolor[rgb]{ .573,  .816,  .314}-0.033 & \cellcolor[rgb]{ .573,  .816,  .314}0.142 & \cellcolor[rgb]{ .573,  .816,  .314}0.018 \\
\cline{2-13}          & AOD   & \cellcolor[rgb]{ .573,  .816,  .314}0.512 & \cellcolor[rgb]{ .753,  0,  0}0.688 & \cellcolor[rgb]{ .753,  0,  0}0.588 & \cellcolor[rgb]{ .753,  0,  0}0.603 & \cellcolor[rgb]{ .753,  0,  0}0.659 & \cellcolor[rgb]{ .573,  .816,  .314}0.353 & \cellcolor[rgb]{ .573,  .816,  .314}0.041 & \cellcolor[rgb]{ .573,  .816,  .314}0.026 & \cellcolor[rgb]{ .573,  .816,  .314}0.020 & \cellcolor[rgb]{ .573,  .816,  .314}0.105 & \cellcolor[rgb]{ .753,  0,  0}0.062 \\
\cline{2-13}          & DI    & \cellcolor[rgb]{ .753,  0,  0}-2.528 & \cellcolor[rgb]{ .753,  0,  0}-2.600 & \cellcolor[rgb]{ .753,  0,  0}-2.468 & \cellcolor[rgb]{ .753,  0,  0}-2.173 & \cellcolor[rgb]{ .753,  0,  0}-2.111 & \cellcolor[rgb]{ .573,  .816,  .314}-1.747 & \cellcolor[rgb]{ .573,  .816,  .314}-0.076 & \cellcolor[rgb]{ .573,  .816,  .314}-1.123 & \cellcolor[rgb]{ .573,  .816,  .314}-1.080 & \cellcolor[rgb]{ .573,  .816,  .314}1.065 & \cellcolor[rgb]{ .753,  0,  0}0.415 \\
\cline{2-13}          & TI    & \cellcolor[rgb]{ .753,  0,  0}0.131 & \cellcolor[rgb]{ .753,  0,  0}0.155 & \cellcolor[rgb]{ .573,  .816,  .314}0.089 & \cellcolor[rgb]{ .753,  0,  0}0.117 & \cellcolor[rgb]{ .753,  0,  0}0.130 & \cellcolor[rgb]{ .882,  .882,  .882}0.155 & \cellcolor[rgb]{ .573,  .816,  .314}0.105 & \cellcolor[rgb]{ .753,  0,  0}0.189 & \cellcolor[rgb]{ .753,  0,  0}0.203 & \cellcolor[rgb]{ .753,  0,  0}0.070 & \cellcolor[rgb]{ .753,  0,  0}0.036 \\
    \hline
    \end{tabular}}%
  \label{Soft vs Hard}%
\end{table}%

\textbf{Finding: Are voting ensembles guaranteed to be at least as fair as the individual learners?}

In homogeneous models, especially bagging, we have seen that the ensemble is guaranteed to be more fair than the most biased individual learner in the ensemble. However, does that claim hold for heterogeneous models? Are heterogeneous models guaranteed to outperform their individual classifiers? We find that this is not true always true. For example, TM-VC5 is a soft voting ensemble with 3 classifiers \textit{Logistc Regression (LR)}, \textit{Random Forest (RF)} and \textit{AdaBoost (ADB)}. The results for these individual classifiers along with the ensemble are reported in Table \ref{Voting-Guarantee}. We can see that in terms of \textit{SPD} and \textit{AOD}, the Voting ensemble actually performs worse than the most biased individual classifiers (\textit{LR} in this case). Recall from earlier that soft-voting takes the average of the probabilities for each class label (with the higher winning). Since the performance in Hard voting is even worse, we can conclude that the performance is not worse due to any single classifier. However, as seen in Table \ref{Voting-Guarantee}, this can be attributed to the fact that the ensemble has outperformed its individual classifiers, in terms of accuracy. In other words, this suggests that the number of correct but \textit{biased} predictions in the ensemble increases (both average of probabilities and majority voting), compared to the total sum of unique biased predictions from the individual classifiers. Upon further investigation, we could not specifically pinpoint responsibility to a single classifier rather than the combination of them. When we added further classifiers like XGBoost and Bagging Classifier, we saw the ensemble outperform significantly (SPD: -0.685 \& AOD: 0.518). Since there is no method to select classifiers before testing/validating the ensemble, we suggest using at-least 5 classifiers in the voting ensemble, which is the next odd number, to increase the diversity of the model pool and reduce chances of biased performance. We note that all the other Titanic models do not exhibit this phenomenon and have more than 3 individual classifiers which reduces the probability of stumbling upon such a combination of probabilities.

\begin{table}[htbp]
  \centering
  \caption{Performance of individual classifiers vs final voting ensemble TM-VC5}
    \begin{tabular}{|c|cccc|}
    \hline
    \multicolumn{1}{|c}{} & \multicolumn{1}{|c}{\textbf{LR}} & \textbf{RF} & \textbf{ADB} & \textbf{Voting} \\ \hline
    \textbf{Acc} & 0.795 & 0.787 & 0.761 & 0.810 \\
    \textbf{F1} & 0.718 & 0.698 & 0.686 & 0.738 \\
    \textbf{SPD} & -0.751 & -0.729 & -0.722 & -0.772 \\
    \textbf{EOD} & -0.671 & -0.671 & -0.671 & -0.625 \\
    \textbf{AOD} & 0.628 & 0.614 & 0.621 & 0.629 \\
    \textbf{DI} & -2.074 & -2.146 & -1.773 & -2.099 \\
    \textbf{TI} & 0.137 & 0.148 & 0.144 & 0.124 \\
    \hline
    \end{tabular}%
  \label{Voting-Guarantee}%
\end{table}%

\textbf{Finding: Odd number of classifiers in hard voting classifiers?}

\begin{mdframed}[leftmargin=1cm,
    skipabove=0.2cm,
    linecolor=white,
    backgroundcolor=gray!5,
    linewidth=2pt,
    roundcorner = 5pt,
    innerleftmargin=1em,
    innerrightmargin=1em,
    innertopmargin=.75em,
    innerbottommargin=.75em,

]
\noindent \textbf{RQ3. How does increasing or decreasing the number of weak learners affect fairness in ensemble learning?}
\end{mdframed}

As discussed before, ensemble methods are mainly divided into homogeneous and heterogeneous models. \textit{Number of trees} or \textit{number of boosting rounds} is one of the most important hyper-parameters for homogeneous methods and refers to the total number of individual learners that make up the ensemble. To understand the composition of fairness, the effect of the \textit{number of trees/rounds} on fairness in Boosting and Bagging methods needs to be explored.In this section, we discuss the implication of number of inidividual learners on fairness.

\textbf{Boosting.} We analyze the boosting models (XGBoost, AdaBoost and Gradient Boosting) across the four datasets in our benchmark to investigate the effect of number of boosting rounds on fairness. We generate the fairness metrics and accuracy scores after each boosting round, and analyze the fairness performance across different models. We have found that given a biased dataset against a protected attribute, the number of weak boosting rounds overall has a positive affect on fairness performance metrics such as SPD, EOD and DI but negatively impacts AOD. For the datasets in our benchmark, the statistical parity difference (SPD) score for Titanic dataset is -0.553, which indicates a higher level of bias against male passengers. On the other hand, Adult, Bank Marketing and German Credit have a SPD score of -0.196303, 0.130428, -0.074801 respectively, that are much lower. To understand this better, let's take a look at Figure \ref{titanic_boosters} which shows the performance of boosting models for the Titanic dataset.

\begin{figure}[ht]
\footnotesize

    \centering
    \includegraphics[width=\linewidth]{Figures/Titanic_Boosting_Estimators.pdf}

  \caption{Performance of boosting models in the Titanic Dataset across boosting round (DI dropped for scaling) }
  \label{titanic_boosters}
\end{figure}

We can see that for all of the different boosting models, except TM-ADB2, SPD, AOD, EOD improve as the boosting rounds proceed. This is due to the behavior of weak learners and the design of boosting models to improve upon the previous learner's incorrect predictions. Therefore, when a dataset is very biased against the protected attribute in an unbalanced dataset, the protected attribute is one of the strongest feature in the weak learner. Since most of the predictive ability of the model comes from the protected attribute, boosting tries to improve performance by using other features and hence, reducing the bias in the final model. For XGB and ADB models, the default number of learners (100) is enough to reach convergence on fairness score. Models TM-XGB2, XGB3 and ADB1, use GridSearch to tune the models and have been assigned very high number of estimators. However, there is no  effect on fairness performance beyond the initial default 100 learners, and is very computationally expensive. On the other hand, GBC models show improvement in fairness when the number of learners are increased beyond. Lastly, our results show that improvement in fairness across boosting rounds does not negatively impact the accuracy, which is interesting given the usual fairness and accuracy trade-off.

\begin{table}[htbp]
  \centering
  \footnotesize
  \caption{Performance of \textit{AC-ADB1} (AdaBoost) across boosting rounds}
  \renewcommand{\tabcolsep}{1pt}
    \begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|c|c|}
    \hline
          & \textbf{classifier} & \textbf{T0} & \textbf{T10} & \textbf{T20} & \textbf{T30} & \textbf{T40} & \textbf{T50} & \textbf{T60} & \textbf{T70} & \textbf{T80} & \textbf{T90} & \textbf{T99} \\
    \hline
    \textbf{AC} & 0.832 & 0.813 & 0.833 & 0.830 & 0.835 & 0.837 & 0.833 & 0.835 & 0.834 & 0.833 & 0.831 & 0.832 \\
    \hline
    \textbf{F1} & 0.627 & 0.605 & 0.629 & 0.627 & 0.628 & 0.634 & 0.629 & 0.630 & 0.634 & 0.623 & 0.624 & 0.627 \\
    \hline
    \textbf{SPD} & -0.204 & -0.183 & -0.178 & -0.173 & -0.185 & -0.188 & -0.198 & -0.191 & -0.200 & -0.192 & -0.202 & -0.204 \\
    \hline
    \textbf{EOD} & -0.179 & -0.063 & -0.077 & -0.065 & -0.117 & -0.102 & -0.143 & -0.143 & -0.119 & -0.165 & -0.159 & -0.179 \\
    \hline
    \textbf{AOD} & 0.140 & 0.081 & 0.081 & 0.073 & 0.103 & 0.097 & 0.121 & 0.117 & 0.111 & 0.128 & 0.131 & 0.140 \\
    \hline
    \textbf{DI} & -1.310 & -0.960 & -1.075 & -0.980 & -1.176 & -1.180 & -1.253 & -1.202 & -1.239 & -1.247 & -1.289 & -1.310 \\
    \hline
    \textbf{TI} & 0.127 & 0.131 & 0.127 & 0.126 & 0.128 & 0.125 & 0.127 & 0.127 & 0.124 & 0.129 & 0.128 & 0.127 \\
    \hline
    \end{tabular}%
  \label{AC-ADB1}
\end{table}%

For the other datasets, we did not see substantial improvement or change in fairness performance with more boosting rounds.  Since the datasets are less biased, the weak learners do not derive the majority of the predictive ability from the protected attributes and as boosting is designed to improve accuracy performance, the fairness metrics converge accordingly. In other words, the weak learners do not show the same high level of bias as the models in Titanic. For instance, we analyzed \textit{AC-ADB1} and found that the final fairness performance is worse than if the boosting round had been stopped early, for very similar accuracy performance (accuracy \& f1). Table \ref{AC-ADB1} shows the performance across boosting rounds.

From the results, we can see that after $10^{th}$ and $20^{th}$ boosting rounds, the model outperforms the final classifier result in both accuracy and fairness metrics. On further inspection, we found that the difference between the predictions of the final classifier and T10 was limited to only 31 instances, all of which were \textit{Females} (protected group) who had been predicted correctly \& positively (\textit{Income} $>$ 50K). Since, boosting was designed to improve performance, the final fairness results can vary. This phenomenon was seen in all the other models and datasets. However,
this is a very interesting finding because it provides further insight into boosting models and how they can be leveraged to tweak boosting rounds to achieve better fairness while being very accurate.

\textbf{Bagging}

Compared to boosting models, bagging models exhibit a very different behavior as number of individual learners are increased. For the bagging models (RF,ET) in our benchmark, the results show that a higher number of learners does not have any effect on fairness performance. For all the models across our datasets, we see the number of learners smoothen the performance to avoid extremes. Though this guarantees that the final model is fair than the most biased model, it also interestingly indicates that using high number of trees is computationally expensive and not effective, empirically. To this end, we conclude that the default number of learners (100), adequately satisfy, not only accuracy performance, but also fairness.  \textbf{[Discuss how to represent this result]}

\begin{mdframed}[leftmargin=1cm,
    skipabove=0.2cm,
    linecolor=white,
    backgroundcolor=gray!5,
    linewidth=2pt,
    roundcorner = 5pt,
    innerleftmargin=1em,
    innerrightmargin=1em,
    innertopmargin=.75em,
    innerbottommargin=.75em,

]
\noindent \textbf{RQ5. What specifications of individual learners can be inferred in different ensemble scenario?}
\end{mdframed}

\textit{\textbf{Stacking}.} We analyze the stacking models for the four datasets in our benchmark. For Titanic ML dataset, we had enough kernels to select the top-6 based on up-votes. However, for other datasets there were few to no stacking model kernels available. For these datasets, we used the top-voted kernel with multiple models to build a stacking model using the various types of ML models in the kernel. This ensures that our results are consistent across datasets and account for variability in datasets.

Stacking uses a two-level learning method to improve model performance by stacking the first-level individual learner's  and inputting their output  to the second-level meta-learner (default: Logistic Regression in scikit-learn) to generate predictions \cite{WOLPERT1992241}. We analyze the stacking models by generating fairness results not only for the final ensemble model, but also for the chain of individual estimators that are part of the first-level models.

\textbf{\textit{Analysis.}}

\textit{Two-layered estimators improve fairness.} All of the Titanic ML stacking models shown in Figure \ref{stacking:comparison} exhibit similar bias except \textit{TM-STK5}, which is the least biased model for the Titanic ML dataset for all fairness metric except Thiel Index (TI). On closer inspection, we found out that \textit{TM-STK5} uses a two-layered stacking approach where the second layer of the individual estimators, act as the meta-learner for the first layer:

\definecolor{codegreen}{rgb}{0,0.6,0}
\definecolor{codegray}{rgb}{0.5,0.5,0.5}
\definecolor{codepurple}{rgb}{0.58,0,0.82}
\definecolor{backcolour}{rgb}{0.95,0.95,0.92}

\lstdefinestyle{mystyle}{
    backgroundcolor=\color{backcolour},
    commentstyle=\color{codegreen},
    keywordstyle=\color{magenta},
    numberstyle=\tiny\color{codegray},
    stringstyle=\color{codepurple},
    basicstyle=\ttfamily\footnotesize,
    breakatwhitespace=false,
    breaklines=true,
    captionpos=b,
    keepspaces=true,
    numbers=left,
    numbersep=5pt,
    showspaces=false,
    showstringspaces=false,
    showtabs=false,
    tabsize=2
}
\lstset{style=mystyle}

\begin{lstlisting}[language=Python]
layer_one_estimators = [('rf_1', RandomForestClassifier(n_estimators=40, random_state=42)),('knn_1', KNeighborsClassifier(n_neighbors=6))]
layer_two_estimators = [('rf_2', RandomForestClassifier(n_estimators=40, random_state=42)),('xg_2', XGBClassifier(objective = 'binary:logistic', colsample_bytree = 0.8, learning_rate = 0.3, max_depth = 7, min_child_weight = 3, n_estimators = 100, subsample = 0.6))]
layer_two = StackingClassifier(estimators=layer_one_estimators, final_estimator=XGBClassifier(n_estimators  = 100))
model =  StackingClassifier(estimators=layer_one_estimators, final_estimator=layer_two)}
\end{lstlisting}

We investigated why this particular model performed significantly better in fairness terms by comparing it to other kernels for the Titanic ML dataset. We found that the difference in feature set or the combination of individual learners was not responsible, rather the better fairness performance is attributed to the two-layered nested stacking approach.

\begin{table}[!htbp]
  \centering
  {\footnotesize

  \caption{\textbf{Fairness metrics of the models using single vs nested-layer stacking}}
  \label{tab:Nested_Stacking}%
 \resizebox{\columnwidth}{!}{   \begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|}
    \hline
    \multirow{2}{*}{\textbf{Models}} & \multicolumn{7}{c|}{\textbf{Single-layer Stacking}}   & \multicolumn{7}{c|}{\cellcolor[HTML]{EBEBEB}\textbf{Nested-layer Stacking}} \\
\cline{2-15}          & \multicolumn{1}{l|}{\textbf{Acc}} & \multicolumn{1}{l|}{\textbf{F1}} & \multicolumn{1}{l|}{\textbf{SPD}} & \multicolumn{1}{l|}{\textbf{EOD}} & \multicolumn{1}{l|}{\textbf{AOD}} & \multicolumn{1}{l|}{\textbf{DI}} & \multicolumn{1}{l|}{\textbf{TI}} & \multicolumn{1}{l|}{\cellcolor[HTML]{EBEBEB}\textbf{Acc}} & \multicolumn{1}{l|}{\cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}\textbf{F1}}} & \multicolumn{1}{l|}{\cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}\textbf{SPD}}} & \multicolumn{1}{l|}{\cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}\textbf{EOD}}} & \multicolumn{1}{l|}{\cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}\textbf{AOD}}} & \multicolumn{1}{l|}{\cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}\textbf{DI}}} & \multicolumn{1}{l|}{\cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}\textbf{TI}}} \\
    \hline
    \multicolumn{1}{|l|}{TM-STK1} & 0.802 & 0.741 & -0.717 & -0.639 & 0.554 & -2.407 & 0.178 & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.765} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.701} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}-0.697} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}-0.613} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.598} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}-2.071} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.196} \\
    \hline
    \multicolumn{1}{|l|}{TM-STK2} & 0.847 & 0.800 & -0.843 & -0.720 & 0.675 & -2.642 & 0.114 & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.728} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.660} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}-0.454} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}-0.392} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000} 0.300} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}-1.152} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.209} \\
    \hline
    \multicolumn{1}{|l|}{TM-STK3} & 0.836 & 0.753 & -0.773 & -0.617 & 0.610 & -2.398 & 0.117 & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.772} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.630} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}-0.488} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}-0.367} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.347} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}-1.612} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.184} \\
    \hline
    \multicolumn{1}{|l|}{TM-STK4} & 0.828 & 0.763 & -0.781 & -0.623 & 0.625 & -2.468 & 0.113 & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.757} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.663} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}-0.486} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}-0.480} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.349} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}-1.305} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.163} \\
    \hline
    \multicolumn{1}{|l|}{TM-STK6} & 0.877 & 0.836 & -0.756 & -0.645 & 0.504 & -2.315 & 0.096 & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.836} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.790} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}-0.542} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}-0.356} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.255} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}-1.329} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.111} \\
    \hline
    AC-STK1 & 0.853 & 0.662 & -0.184 & -0.103 & 0.090 & -1.289 & 0.122 & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.809} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.532} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}-0.104} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}-0.009} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.022} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}-0.737} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.165} \\
    \hline
    BM-STK1 & 0.910 & 0.549 & 0.123 & 0.095 & 0.078 & 0.895 & 0.070 & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.896} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.430} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.101} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}-0.007} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.046} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.901} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.087} \\
    \hline
    GC-STK1 & 0.680 & 0.342 & -0.126 & -0.045 & 0.142 & -0.152 & 0.115 & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.627} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.423} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}-0.074} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}-0.121} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.074} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}-0.108} & \cellcolor[HTML]{EBEBEB}{\color[HTML]{000000}0.233} \\
    \hline
    \end{tabular}}}%
\end{table}%

We validated our findings by training all of the stacking based kernels in our benchmark on this 2-layered nested stacking model. To ensure consistency, we did not change the feature set or any preprocessing method for any kernel. The results are shown in Table \ref{tab:Nested_Stacking}. For all stacking models across the four datasets in our benchmark, every model significantly improved in all fairness measures while not affecting accuracy measures drastically. For example, \textit{TM-STK6} improved \textbf{Statistical Parity Difference} (SPD) scores by $28\%$ while only accuracy dropping $4.68\%$. Overall across all datasets, the accuracy score dropped by $6.50\%$ while the SPD improved by $31.8\%$. Two-layered nested stacking approach improves fairness scores dramatically due to the addtion of a second-layer of ensembles that act as a meta-learner. Stacking generalization works by first estimating the error of the generalizer and then using a second classifier (meta-learner) to \textit{correct} the errors i.e. whose output is the final prediction \cite{WOLPERT1992241}. In two-layered stacking, the first layer uses a list of estimators (ensembles) to generate a set of predictions for each of the estimators, instead of using a single classifer (meta-learner), the predictions are fed into another set of estimators. This ensures that a single meta-learner does not entirely predict the final outcome based on the first-level errors. Rather the nested ensembles create a new set of predictions which are then finally inputted into the final meta-classifier of the second layer. The introduction of a second-layer of ensembles guarantees that the final fairness metrics will be better than the most biased single classifier \cite{Nina}.

\textit{Default meta-learner optimized for accuracy.} The default meta-learner for stacking models in Sklearn is logistic regression. Our results show that while the accuracy of the models are better with logistic regression as the meta-learner, it introduces significant bias in the final predictions. We analyze all the models that use \textit{LogisticRegression()}: \textit{TM-STK2, TM-STK4, TM-STK6, AC-STK1, BM-STK1, GC-STK1}. For all these models, we use XGBoost as the meta-learner instead, as an example of an ensemble method that performs equally well but is less biased. Our results did not significantly vary between the use of other ensembles as the final estimator. The change in performance and total bias (sum of the absolute values of bias) is depicted in Figure \ref{stacking:comparison}. As we can see, the total bias is reduced for all the stacking models in our benchmark while there is very minimal change in accuracy scores. The maximum change in performance for any of the model is less than 0.1 (GC-STK). Therefore, the use of an ensemble as a final estimator can improve fairness of stacking models.

\begin{figure}[!htbp]
  \centering

    \includegraphics[scale=0.4]{Figures/Finding2_stacking.pdf}
    \caption{Change of performance and total bias when using XGBoost Classifier as meta-learner in Stacking (negative value denotes reduction}
    \label{stacking:comparison}
\end{figure}

\textbf{AdaBoost.}

\lstset{style=mystyle}

\begin{lstlisting}[language=Python]
model = AdaBoostClassifier(DecisionTreeClassifier())
model = AdaBoostClassifier(base_estimator = DecisionTreeClassifier(criterion = 'entropy'), learning_rate = 0.3, random_state = 7)
\end{lstlisting}

\textbf{Analysis.}

\textit{Decision stumps vs deep trees.} The default base estimator for AdaBoost is a decision tree classifier with \textit{max depth = 1}, also called a decision stump. Decision stump is a decision tree with only 1 split, which is usually the best split based on the training features. In our benchmark, \textit{AC-ADB2} and \textit{TM-ADB1} were the only two models that did not use the default decision stump. To investigate the effect of using a decision stump with a full-grown tree on fairness, we compare the performance of both of these models using these two methods. As shown in \ref{Fig:decision vs stump}, we see similar behavior to what we saw as boosting rounds progressed. For highly biased dataset like Titanic, decision stumps worsen the behavior as the weak learner is primarily composed of the protected attribute due to it's higher predictive power. However, the opposite is seen in AC-ADB2 because deeper trees are more complex and will add bias by using the predictive power in the protected attribute, which is not present in it's weak learner

\begin{figure}[!htbp]
\centering
\begin{minipage}{.5\textwidth}
  \centering
  \includegraphics[width=.8\linewidth]{Figures/chart-2.pdf}
  \label{fig:test1}
\end{minipage}%
\begin{minipage}{.5\textwidth}
  \centering
  \includegraphics[width=.8\linewidth]{Figures/chart_1.pdf}
  \label{fig:test2}
\end{minipage}
\caption{Percentage change in performance using Decision Stump AC-ADB2 and TM-ADB1}
\label{Fig:decision vs stump}
\end{figure}

\begin{figure}[htbp]
\centering
\begin{minipage}{.5\textwidth}
  \centering
  \includegraphics[width=.8\linewidth]{Figures/TM_RandomSplit.pdf}
  \label{fig:test1}
\end{minipage}%
\begin{minipage}{.5\textwidth}
  \centering
  \includegraphics[width=.8\linewidth]{Figures/AC_RandomSplit.pdf}
  \label{fig:test2}
\end{minipage}
\caption{Percentage change in performance using Decision Stump AC-ADB2 and TM-ADB1}
\label{Fig:decision vs stump}
\end{figure}

\begin{mdframed}[leftmargin=1cm,
    skipabove=0.2cm,
    linecolor=white,
    backgroundcolor=gray!5,
    linewidth=2pt,
    roundcorner = 5pt,
    innerleftmargin=1em,
    innerrightmargin=1em,
    innertopmargin=.75em,
    innerbottommargin=.75em,

]
\noindent \textbf{RQ6. How the fairness composition in ensemble be leveraged towards building fairer models?}
\end{mdframed}

In this RQ, we investigate how we can use the composition methods to build fairer models...

\section{Background and Motivation}
\label{background}

In this section, we first discuss fairness in ML software and some terminologies. Next, we present an ensemble to show that ensemble design affects the fairness of the model and the need to study bias induced by certain ensemble parameters.

\begin{table*}[!htbp]
  \centering
  \footnotesize
  \renewcommand{\tabcolsep}{1pt}

  \caption{Ensemble types used in our experiments}
   \begin{tabular}{c|c|m{6.5cm}|c|c}
    \hline
    \multicolumn{1}{c|}{\cellcolor[HTML]{C0C0C0}\textbf{Categories}} & \multicolumn{1}{c|}{\cellcolor[HTML]{C0C0C0}\textbf{Ensemble Types}} & \multicolumn{1}{c|}{\cellcolor[HTML]{C0C0C0}\textbf{Algorithms}} & \multicolumn{1}{c|}{\cellcolor[HTML]{C0C0C0}\textbf{Composition}} & \multicolumn{1}{c}{\cellcolor[HTML]{C0C0C0}\textbf{Classifiers}} \\
    \hline
    \multirow{3}{*}{Sequential} & \multirow{3}{*}{Boosting} & \multirow{3}{*}{\shortstack {Construct n homogeneous estimators sequentially that
    \\ improve predictions based on the previous estimator's
    \\ incorrect predictions}} & Homogeneous & XGBoost \\
          &       &       & Homogeneous & AdaBoost \\
          &       &       & Homogeneous & Gradient Boosting \\
    \hline
    \multirow{5}{*}{Parallel} & \multirow{3}{*}{Bagging} & \multirow{3}{*}{\shortstack{Construct \textit{n} parallel homogeneous models that are
    \\ aggregated using averaging}} & Homogeneous & Random Forest \\
         &       &       & Homogeneous & ExtraTrees \\
         &       &       & Homogeneous & Bagging Classifier \\
\cline{2-5}          & Voting & Construct a list of n hetereogeneous user-specified weighted classifiers that are aggregated using majority voting or argmax & Heterogeneous & Voting Classifier \\
\cline{2-5}          & Stacking & Construct a list of n hetereogeneous classifiers as base learners and a meta-classifier to decide weights for each learner & Heterogeneous & Stacking Classifier \\
    \hline
    \end{tabular}%
  \label{tab:ensemble-types}%
\end{table*}%

\begin{table}[htbp]
  \centering
  \scriptsize
  \renewcommand{\tabcolsep}{1pt}
  \caption{Summary of the datasets and the number of models collected for each in the benchmark }
    \begin{tabular}{c|c|c|c|c|c|c|c|c|c|c}
    \hline
    \cellcolor[HTML]{C0C0C0}\textbf{Datasets} & \cellcolor[HTML]{C0C0C0}\textbf{PA} &
    \cellcolor[HTML]{C0C0C0}\textbf{Size} &
    \cellcolor[HTML]{C0C0C0}\textbf{\# XGB} & \cellcolor[HTML]{C0C0C0}\textbf{\# ADB} & \cellcolor[HTML]{C0C0C0}\textbf{\# GBC} & \cellcolor[HTML]{C0C0C0}\textbf{\# RF} & \cellcolor[HTML]{C0C0C0}\textbf{\# ET} & \cellcolor[HTML]{C0C0C0}\textbf{\# STK} & \cellcolor[HTML]{C0C0C0}\textbf{\# VT} & \cellcolor[HTML]{C0C0C0}\textbf{Total} \\
    \hline
    \cellcolor[HTML]{EDEDED}\textbf{Adult Census} & \cellcolor[HTML]{EDEDED} sex &
    \cellcolor[HTML]{EDEDED} 32561 &
    \cellcolor[HTML]{EDEDED} 6     & \cellcolor[HTML]{EDEDED} 6     &  \cellcolor[HTML]{EDEDED} 6     & \cellcolor[HTML]{EDEDED} 6     &  \cellcolor[HTML]{EDEDED} 6     &  \cellcolor[HTML]{EDEDED} 1     & \cellcolor[HTML]{EDEDED} 2     & \cellcolor[HTML]{EDEDED} 33 \\
    \textbf{Titanic ML} & sex
    & 891 & 6     & 6     & 6     & 6     & 6     & 6     & 6     & 42 \\
    \cellcolor[HTML]{EDEDED}\textbf{Bank Marketing} & \cellcolor[HTML]{EDEDED} age   &
    \cellcolor[HTML]{EDEDED} 41118 &
    \cellcolor[HTML]{EDEDED} 6     & \cellcolor[HTML]{EDEDED} 6     & \cellcolor[HTML]{EDEDED} 6     & \cellcolor[HTML]{EDEDED} 6     & \cellcolor[HTML]{EDEDED} 2     &  \cellcolor[HTML]{EDEDED} 1     & \cellcolor[HTML]{EDEDED} 2     & \cellcolor[HTML]{EDEDED} 29 \\
    \textbf{German Credit} & sex   & 1000 & 6     & 1     & 1     & 6     & 1     & 1     & 1     & 17 \\
    \hline
    \end{tabular}%
  \label{tab:datasetsummary}%
  \begin{tablenotes}
      \small
      \centering
      \item  XGB: XGBoost, ADB: AdaBoost, GBC: Gradient Boosting
      , RF: Random Forest, ET: Extra Trees, STK: Stacking, VT: Voting
    \end{tablenotes}
\end{table}%

\subsection{Ensemble learning in ML software}
\label{subsec:ensemble-in-ml}
{\em Ensemble models} are a class of ML classifiers where the predictions from different learners (models) are pooled using a combination method (voting, average, random etc) to make the final predictions.
\textcolor{blue}{Ensemble models are the most mentioned ML/DL algorithms on Kaggle \cite{DBLP:journals/corr/abs-2006-08334} and are widely used due to their superior performances in ML challenges \cite{10.5555/2627435.2697065, inproceedings-OnlineLearning, dietterich2000experimental, bauer1999empirical, HOSNI201989}. Out of the top models on Kaggle for popular fairness datasets used by previous SE researchers, ensemble models comprise more than $80\%$ of the total models \cite{10.1145/3368089.3409704,10.1145/3468264.3468536}. Just like traditional ML models, we observe that ensembles also suffer from fairness issues. Categories of ensembles are based on homogeneity, learning technique, and ensemble types. All the ensemble types covered in our study and the corresponding classifiers are given in Table \ref{tab:ensemble-types}. There are mainly two categories of ensemble models: Sequential and Parallel.}

\textbf{\textit{Sequential Ensembles.}} These ensembles sequentially generate base learners. Each learner in the ensemble depends on the previous learners in the sequence because the next learner attempts to correct the wrong predictions from the previous learner and so on \cite{8360836}. AdaBoost is an example of a sequential model where it reweighs (higher) misclassified examples.

\textbf{\textit{Parallel Ensembles.}} Parallel ensembles train individual base learners in parallel and independent of each other. A random subset of training data is used to construct these learners and combined using techniques such as bagging or voting, which encourages improved variance \cite{8360836} e.g., Random Forest.

\textit{\textbf{Homogeneity of ensembles.}}
Ensemble methods that use single-type base learners are called \textit{homogeneous} models \cite{Elish2013EmpiricalSO}. These individual learners are combined to generate the final result, e.g., {\em XGBoost} and {\em AdaBoost} use decision trees. By contrast, \textit{heterogeneous} ensembles combine the predictions of disimilar individual learners \cite{Elish2013EmpiricalSO}. A popular heterogeneous ensemble method is {\em Voting}. Finally, ensemble method types are divided into {\em Boosting, Bagging, Voting}, \& {\em Stacking} \cite{scikit-learn}.

\textcolor{blue}{In this paper, we use the term {\em ensemble parameters} to mean hyperparameters like \textit{number of learners}, \textit{combination methods}, etc., that dictate how ensembles combine base learners and hence affect ensemble fairness \& design (Table \ref{Parameters}).}

\subsection{Motivation}
\textcolor{blue}{Unlike traditional software, data-driven software takes input data, performs pre-processing, and applies an ML algorithm with tuned hyperparameters to build a rule-based logic that a developer does not write manually. These hyperparameters are tuned for accuracy and have been shown to encode bias \cite{}. However, no work has studied the composition of fairness in ensembles and the bias induced by \textit{ensemble hyperparameters}. In this study, we postulate that the ensembles can be designed to be fairer without using any mitigation techniques and explore fairness composition to design fair ensembles software.}

\section{Conclusion}
\label{sec:conclusion}

Ensembles are widely used for predictive tasks due to superior performance. However, most approaches to measuring fairness and mitigation focus on single classifiers. In this paper, we conduct an empirical study to evaluate the composition of fairness in popular ensemble techniques. The results showed that base learners induce bias in ensembles and that we can mitigate inherent bias in ensembles by using certain base learner configurations and appropriate parameters. Lastly, works have shown the need to support developers during model training in mitigating bias. Our analysis of the hyperparameter space should help developers build fairness-aware ensembles and automated tools to detect bias in ensembles.
\section{Discussion}
\label{sec:discussion}

In this study, we undertook an important task of understanding the composition of fairness in ensemble learning that has gained popularity recently \cite{10.1145/3368089.3409704,10.1145/3468264.3468536}. However, most works focus on fairness mitigation in single classifiers, which fail to capture inherent bias in ensembles. Our work has shown that fair ensembles can be designed without mitigation techniques. Our research also identifies unfairness causes in different ensembles and their interplay with the input space, which would guide fairness bug localization and repair in ensemble learning. For example, we reported fairness patterns in individual learners that can induce bias in ensembles such as tree depth, min leaf node samples, etc. These can also be leveraged for fairness-improving interventions such as feature selection, data preprocessing, etc. Overall, our result would draw attention to the fairness of ensembles which are popular learning algorithms but mostly overlooked by the community.

Moreover, research in SE showed the impact of hyperparameters on fairness, and their role in helping developers mitigate bias during model training \cite{10.1145/3510003.3510202, 10.1145/3468264.3468536, 10.1145/3368089.3409697}. We extend that to explore the hyperparameter space for ensembles to guide developers to design fair ensembles using currently available composition and configuration. We discussed the effect of regularization on fairness which should encourage the development of fairness-aware regularization techniques and investigation of the trade-off between fairness mitigation and overfitting, similar to the accuracy-fairness trade-off. We found that many ensemble models do not have library support to monitor the fairness of individual learners. Our work would encourage the development of tools and API support to improve the transparency of ML software to address fairness concerns.

\section{Fairness Guarantees in Ensemble}
\label{sec:fairness-guarantee}

In \S 4, we analyzed the composition of fairness in heterogeneous models which combine the powers of each individual learner to aggregate fairness scores. \citeauthor{Nina} first presented theoretical evidence of an ensemble of random selection based classifiers being fair \cite{Nina}. More recent works provided empirical evidence and built upon this by arguing that the ensemble is guaranteed to be at least as fair as the most biased classifier in the ensemble \cite{8995403, 9462068}. However, these observations were limited to bagging and random selection strategy only. In this section, we explore whether heterogeneous learners guaranteed to outperform their individual learners \textbf{(RQ2)} and provide empirical evidence to the contrary.

We explore this for both stacking and voting and find that this is not always true. For example, TM-VC5 is a soft voting ensemble with 3 classifiers \textit{Logistic Regression (LR)}, \textit{Random Forest (RF)} and \textit{AdaBoost (ADB)}. The results for these individual classifiers along with it's ensemble are reported in Table \ref{Voting-Guarantee}. We can see that in terms of \textit{SPD} and \textit{AOD}, the Voting ensemble actually performs worse than the most biased individual classifiers (\textit{LR} in this case). Recall from earlier that soft-voting takes the average of the probabilities for each class label (with the higher winning). Upon further investigation, we find that that the ensemble has outperformed its individual classifiers, in terms of accuracy. In other words, this suggests that the number of correct but \textit{biased} predictions in the ensemble increases. Therefore, this shows that classifiers of voting ensembles can combine in a way that can worsen the fairness performance of the ensemble for specific metrics.

Similarly, we see that stacking is not guaranteed to outperform  e.g. AC-STK1. As illustrated in Table \ref{Voting-Guarantee}, AC-ST1 has 5 estimators \textit{ExtraTrees (ET), Gradient Boosting Classifier (GBC), Random Forest (RF), XGBoost (XGB) and Light Gradient Boosting (LGBM)}. We can see that the final stacking \textit{SPD} score is slightly worse than \textbf{GBC}, which is the most biased estimator in stacking. We do note that the difference here is insignificant, however, it provides empirical evidence of existence of combination that can potentially result in a more biased ensemble.

\begin{table}[t]
  \centering
  \footnotesize
 \caption{Comparison of fairness performance between individual classifiers and final voting \& stacking classifiers. For SPD and AOD, TM-VT5 is more bias than LR while AC-STK1 is more unbiased than GBC  (worst learners in each ensembles)}
\resizebox{\columnwidth}{!}{
    \begin{tabular}{|c|c|cccccc|}
    \hline
    \multirow{8}{*}{\rotatebox[origin=c]{90}{TM-VT5}} &  & {\textbf{LR}} & \textbf{RF} & \textbf{ADB} & \textbf{Voting} &       &  \\ \hline
          & \textbf{Acc} & 0.795 & 0.787 & 0.761 & 0.810 &       &  \\
          & \textbf{F1} & 0.718 & 0.698 & 0.686 & 0.738 &       &  \\
          & \textbf{SPD} & -0.751 & -0.729 & -0.722 & -0.772 &       &  \\
          & \textbf{EOD} & -0.671 & -0.671 & -0.671 & -0.625 &       &  \\
          & \textbf{AOD} & 0.628 & 0.614 & 0.621 & 0.629 &       &  \\
          & \textbf{DI} & -2.074 & -2.146 & -1.773 & -2.099 &       &  \\
          & \textbf{TI} & 0.137 & 0.148 & 0.144 & 0.124 &       &  \\
          \hline \hline
     \multirow{8}{*}{\rotatebox[origin=c]{90}{AC-STK1}} &  & \textbf{ET} & \textbf{GBC} & {\textbf{RF}} & {\textbf{XGB}} & {\textbf{LGBM}} & \textbf{Stacking} \\ \hline
          & \textbf{Acc} & 0.842 & 0.853 & 0.842 & 0.851 & 0.851 & 0.853 \\
          & \textbf{F1} & 0.641 & 0.661 & 0.648 & 0.660 & 0.661 & 0.662 \\
          & \textbf{SPD} & -0.153 & -0.183 & -0.161 & -0.175 & -0.182 & -0.184 \\
          & \textbf{EOD} & -0.020 & -0.108 & -0.030 & -0.074 & -0.098 & -0.103 \\
          & \textbf{AOD} & 0.040 & 0.092 & 0.048 & 0.073 & 0.087 & 0.090 \\
          & \textbf{DI} & -0.961 & -1.302 & -0.976 & -1.166 & -1.235 & -1.289 \\
          & \textbf{TI} & 0.128 & 0.122 & 0.124 & 0.122 & 0.121 & 0.122 \\
          \hline
    \end{tabular}}%
  \label{Voting-Guarantee}%
  \vspace{-4mm}
\end{table}%

Overall, our empirical evaluation of heterogeneous models with stacking and voting show that they perform comparably (at best) to the worst classifier in the ensemble but are not necessarily guaranteed to outperform the most biased classifier in terms of fairness. To reduce the possibility of this, developers should add more number of classifiers, especially in soft voting, where a highly biased classifier can skew the final result negatively. This is discussed further in detail in \S \ref{sec:fair-design}.

\section{Introduction}
\label{sec:introduction}

Machine learning (ML) is ubiquitous in modern software today. Due to the black-box \cite{10.1145/3338906.3338937} nature of ML algorithms and its applications in critical decision-making \cite{10.1145/3278721.3278729,mattu_machine_nodate}, fairness in ML software has become a huge concern. Measuring ML fairness \cite{10.1145/2090236.2090255, three-bayes, 10.1145/3219819.3220046, JMLR:v20:18-262} and mitigating the discrimination \cite{Chouldechova2017FairPW, three-bayes, NIPS2016_dc4c44f6}  has been studied extensively. Recent work in software engineering has shown the need to produce fair software and detect bias in complex ML software environments \cite{fairnessinsoftware, 10.1145/3287560.3287589, 10.1145/3468264.3468536, 10.1145/3368089.3409704}.

Prior research has mostly focused on fairness in standalone classifiers (e.g., \textit{Logistic Regression, SVM}) \cite{10.1145/3338906.3338937, 10.1145/3106237.3106277, article}.  However, a class of ML models called {\em ensemble models} are becoming increasingly important in practice today due to their superior performance across a multitude of ML \& real-life challenges \cite{10.5555/2627435.2697065, oza2001online, dietterich2000experimental, bauer1999empirical, HOSNI201989}, and better generalization on unseen data, especially in smaller datasets \cite{548872, 9364928,dietterich2000experimental}. Ensemble models combine the predictions of multiple base learners to make the final prediction, e.g., Random Forest uses a large number of decision trees, with the majority class the final output. Ensemble models are the most mentioned ML algorithms on Kaggle \cite{DBLP:journals/corr/abs-2006-08334} and in previous SE works on fairness, ensemble models comprise more than $80\%$ of the total models \cite{10.1145/3368089.3409704,10.1145/3468264.3468536}.
Like traditional ML models, ensemble models can also suffer from unfairness problem that discriminates against population subgroups based on race, gender, etc. Although many fairness mitigation techniques \cite{10.1145/3278721.3278779, 10.1145/3368089.3409697} exist, they do not always generalize well \cite{10.1145/2783258.2783311, NIPS2016_9d268236, IBMLale}. Therefore, if we better understand the fairness composition in ensembles, we can design fair ensemble models without applying mitigation techniques. In this paper, we have conducted an empirical study to understand the composition of fairness in ensembles and the interplay of their properties with fairness.

Recently, multiple works have shown that ensembles can be leveraged to enhance fairness and mitigate bias in ML models \cite{9462068, 8995403, IBMLale}. Grgic-Hlaca \MakeLowercase{\textit{et al.}} first explored fairness properties of random selection ensemble, only theoretically \cite{Nina}. Bower \MakeLowercase{\textit{et al.}} explored how fairness propagates through a multi-stage decision process like hiring \cite{article}. Similarly, Dwork \MakeLowercase{\textit{et al.}} introduced a framework to understand the composition of fairness in ensembles that \textit{only} utilize AND, OR operators to make a decision, e.g., two credit bureaus' (AND) report a score to determine loan eligibility \cite{10.1145/2090236.2090255}. Feffer \MakeLowercase{\textit{et al.}} studied how ensembles and bias mitigators can be combined using modularity to improve stability in bias mitigation \cite{IBMLale}. Therefore, it is evident that fairness in ensembles and their composition is non-trivial. Moreover, prior works in SE have shown the impact of training processes such as hyperparameter optimization, data transformation, etc., on the fairness of ML software \cite{10.1145/3510003.3510202, 10.1145/3468264.3468536, 10.1145/3368089.3409697}. We postulate that ensemble hyperparameters also impact unfairness in ensembles, and failure to study them can amplify bias. However, ensemble hyperparameters are different than typical ML model hyperparameters as they dictate the design of the ensemble, e.g., number of learners, learning method, etc. However, no empirical study has been conducted to understand fairness composition in ensembles and the effect of their hyperparameter space on fairness.
To this end, we have created a benchmark of 168 real-world ensemble models from Kaggle and designed experiments to measure their fairness. We analyze fairness composition in ensemble criteria such as parallel and sequential ensembles, homogeneity of models, and different ensemble methods such as bagging, boosting, voting, stacking, etc., and all the ensemble classifiers available in the popular Scikit-learn \cite{scikit-learn}. Specifically, we answer the following overarching research questions:

\textbf{RQ1:} \textit{What are the fairness measures of various ensemble techniques?} \textbf{RQ2:} \textit{How does fairness compose in ensemble models?} \textbf{RQ3:} \textit{Can ensemble-related hyperparameters be chosen to design fair ensemble models?}

To the best of our knowledge, this is the first work to experimentally evaluate the fairness composition in popular ensembles and elicit fair ensemble design considerations. Our results show that fairness in ensembles composes in the base learners and fair ensemble models can be built by carefully considering the composition. The analyses show which learners cause fairness problems which software developers can leverage to develop frameworks to measure fairness in base learners to encourage transparency. We also identify and explore ensemble-related hyperparameters to design fair ML models for each ensemble type. Lastly, we provide a comprehensive review of fairness composition in ensembles that will help direct future research in the area. Overall, the following are the key contributions of this paper:
\begin{itemize}[leftmargin=*]
    \item Explored fairness composition and its interplay with data characteristics and individual learners to mitigate bias.
    \item Empirically evaluates fairness patterns of popular ML ensemble models.
    \item We identified ensemble design considerations and hyperparameters that would guide developers in fair ensemble design and mitigate inherent unfairness effectively.
    \item A comprehensive fairness benchmark of popular ensembles that leveraged for further research on building fairness-aware ensembles. The benchmark, code and experimental results are available: \url{https://github.com/UsmanGohar/FairEnsemble}
\end{itemize}

The rest of the paper is organized as follows: \S \ref{sec:motivation} describes the motivation of our work and the background on ensembles. In \S \ref{sec:methodology}, we discuss the methodology for our study, including benchmark collection, datasets and fairness and accuracy measures used, and finally, the experiment setup \& design. In \S \ref{sec:fairness-in-ensembles}, we discuss the state of fairness in ensembles \textbf{(RQ1)} and how it composes \textbf{(RQ2)}, \S \ref{sec:fair-design} discusses the design criteria to improve fairness in ensembles \textbf{(RQ3)}. Finally, we discuss the implications of our work in \S \ref{sec:discussion}, threats to validity in \S \ref{sec:threats}, related works in \S \ref{sec:related} and then present the conclusion in \S \ref{sec:conclusion}.
\section{Measures}
\label{measures}

Multiple quantitative fairness and accuracy measures are available to evaluate a model. However, we only focus on the measures that have been previously used in literature \cite{10.1145/3287560.3287589, 10.1145/3368089.3409704}, as they  provide a good representation of metrics. Let $D = (T,S,Y)$ be a dataset where $D$ is the portion of the dataset used for training, $S$ is the protected attribute ($S = 1$, if privileged group, otherwise $S = 0$) and $Y$ is the classification label ($Y = 1$ denotes favorable label, otherwise $Y = 0$). Let $\hat{Y}$ denote the prediction of a ML model. Next we define our measures in terms of these notations.

\subsection{Accuracy Measures}

We evaluate the performance of the models using standard accuracy measures for comparison across datasets and relationship with fairness. For accuracy measures, we use the following two accuracy metrics:

\textit{Accuracy.} To evaluate the overall performance of the model, we use the accuracy measure, $P[\hat{Y} = Y]$.
\begin{center}
    Accuracy = $\frac{(TP+TN)}{TP+TN+FP+FN}$
\end{center}

\textit{F1 Score.} To evaluate performance of the model in relation to both recall and precision, we use the F1 score which is the harmonic mean of precision and recall.

\begin{center}
    F1 = 2 *(Precision * Recall)/Precision + recall)
\end{center}

where Recall = $TP/(TP + FN)$, Precision $ = TP/(TP+FP)$

\subsection{Fairness Measures}

Broadly, fairness metrics are divided into three different categories and multiple fairness metrics have been proposed in literature \cite{Binns2018FairnessIM}. We have selected a subset of these metrics that are representative of the three categories, without being exhaustive. Furthermore, we have followed the recommendations of Friedler \textit{et al.} \cite{10.1145/3368089.3409704} in terms of metrics selection.

\textbf{Group fairness metrics:} \\

\paragraph{Equal Opportunity Difference (EOD):} This is defined by the difference of true-positive rates (TPR) between privileged and unprivileged groups.

\begin{center}
    $EOD = TPR_{up} - TPR_p$
\end{center}
where $TPR = TP/(TP + FN), FPR = FP/(FP + TN)$  \\

\textit{Average Odds Difference (AOD):} This is defined by the mean of false-positive rate (FPR) difference and true-positive rate difference between unprivileged and privileged groups. \cite{NIPS2016_9d268236}

\begin{center}
    $AOD = {(FPR_{up} - FPR_p) + (TPR_{up} - TPR_p)}/2$
\end{center}

\paragraph{Disparate Impact (DI):} This is defined as the ratio of the probability of unprivileged group getting a favorable prediction and the probability of privileged group getting a favorable prediction \cite{JMLR:v20:18-262}

\begin{center}
    $DI = P[\hat{S} = 1| Y = 0]/ P[\hat{S} = 1| Y = 1]$
\end{center}

To improve readability when comparing different metrics in the same table, we convert Disparate Impact (DI) to log scale.

\paragraph{Statistical Parity Difference (SPD):} This is defined similarly to DI but in this case, we use a difference between the probabilities \cite{Calders2010ThreeNB}.

\begin{center}
    $SPD = P[\hat{S} = 1| Y = 0] - P[\hat{S} = 1| Y = 1]$
\end{center}

\subsubsection{Individual fairness metrics:}

\paragraph{Theil Index (TI):} Theil Index measures both the group and individual fairness \cite{10.1145/3219819.3220046}. It is defined using the following equation where $b_i = \hat{s}_i - s_i + 1$

\[   TI = \sum_{i=1}^{n} \frac{b_i}{\Bar{a}} ln \frac{b_i}{\Bar{a}}\]

\section{Methodology}
\label{sec:methodology}
In this section, we discuss the benchmark collection process, the datasets, and fairness and accuracy measures. Finally, we describe the experimental design and setup.

\subsection{Benchmark Collection}
\label{subsec:benchmark-collection}

For our experiments, we collected ensemble models from Kaggle \cite{kaggle} for datasets that have been used in prior fairness literature \cite{10.1145/3468264.3468537, 10.1145/3468264.3468536, IBMLale}. Unlike these works, we only collect ensemble-based models for evaluation. Specifically, we collect all ensemble classifiers available via the popular sci-kit learn library \cite{scikit-learn}. We follow a similar benchmark collection process as in \cite{10.1145/3368089.3409704}. Table \ref{tab:datasetsummary} summarizes the datasets and the classifiers in the benchmark.

For each dataset, we collected Kaggle kernels for each ensemble type in Table \ref{tab:ensemble-types} and classifiers available in scikit-learn. We filter these kernels based on four-step filtering criteria: $1)$ it should contain the predictive model (some kernels focus on data exploration only), $2)$ protected attribute is included in the training data, $3)$ at least five up-votes, and $4)$ ranked by up-votes. We used Kaggle API to collect these models and pass them through the filtering criteria. Finally, we select the top 6 models for each ensemble classifier from each dataset. In total, we created a benchmark of 168 ensembles across four datasets. We could not find certain classifiers on Kaggle for datasets like German Credit. To handle those, we use default models from scikit-learn to ensure we can evaluate across different datasets. The number of models mined is shown in Table \ref{tab:datasetsummary}. Next, we present an overview of the datasets used in our benchmark.

\textit{Adult Census}. The dataset contains income and personal information about individuals \cite{adult-kaggle}. We used \textit{sex} as the protected attribute and \textit{male} as the privileged class. The classification task predicts if a person makes over \$50,000 in annual income.

\textit{Titanic}. The dataset contains passenger data, such as gender, cabin class, etc., and is pre-split into train \& test sets; however, the test set does not contain any instance of a male passenger surviving \cite{titanic-kaggle}. Hence, we only use the training set, with \textit{gender} as the protected attribute and \textit{Female} as the privileged class. The prediction task is whether a passenger survives.

\textit{Bank Marketing}. The dataset contains bank customers' personal information such as age, job type, etc. \cite{bank-marketing-kaggle}. The protected attribute is \textit{age}, where \textit{age $> 25$} is considered privileged class and \textit{age $< 25$} as unprivileged \cite{10.1145/3368089.3409704}. The prediction task determines whether a client will subscribe to a term deposit.

\textit{German Credit}. This dataset contains personal and financial information about individuals who apply for loans at a bank \cite{german-credit}. We used the processed dataset \cite{german-credi-kaggle} since most models in our benchmark used it. This version has nine attributes, such as sex, credit amount, etc. We choose \textit{sex} as the protected attribute and \textit{male} as the privileged class. The prediction task is whether an individual is a credit risk.

\subsection{Measures}
\label{subsec:measures}

Multiple quantitative fairness and accuracy measures are available to evaluate a model. We use measures that have been previously used in literature \cite{10.1145/3287560.3287589, 10.1145/3368089.3409704}. Let $D = (T,S,Y)$ be a dataset where $T$ is the training set, $S$ is the protected attribute ($S = 1$, if privileged group (\textit{p}), else $S = 0$ (\textit{up})) and $Y$ is the classification label ($Y = 1$ if favorable label, else $Y = 0$). Let $\hat{Y}$ denote the prediction of an ML model. Next, we define our measures in terms of these notations.

\subsubsection{Accuracy Metrics}
\label{subsubsec:accuracy-measures}

We evaluate the performance of the models using accuracy and F1 metrics as defined below:
\begin{align*}
    &Accuracy = (true\ positive + true\ negative) / total \\
    &F1 = 2 *(precision * recall)/(precision + recall)
\end{align*}

where recall: $TP/(TP + FN)$, precision: $TP/(TP+FP)$ \\

\subsubsection{Fairness Measures}
\label{subsubsec:fairness-measures}

Broadly, fairness metrics are divided into three categories \cite{Binns2018FairnessIM}. We have selected a subset of these metrics representing the three categories without being exhaustive. Furthermore, we have followed the recommendations of Friedler \textit{et al.} \cite{10.1145/3368089.3409704} in terms of metrics selection.

\noindent \textbf{Group fairness metrics:}
\textit{Group fairness} means similar predictive outcomes for protected attributes, e.g., race (Asian/White) on a group level.

\noindent \textit{Equal Opportunity Difference (EOD):} This is defined as the difference of true-positive rates (TPR) between privileged and unprivileged groups \cite{Bellamy2018AIF3}.
\begin{align*}
    EOD = TPR_{up} - TPR_p
\end{align*}
where TPR: $TP/(TP + FN)$, FPR:  $FP/(FP + TN)$ \\

\noindent \textit{Average Odds Difference (AOD):} This is defined as the mean of false-positive rate (FPR) difference and true-positive rate difference between unprivileged and privileged groups \cite{NIPS2016_9d268236}.

\begin{center}
    $AOD = {[(FPR_{up} - FPR_p) + (TPR_{up} - TPR_p)}]/2$
\end{center}

\noindent \textit{Disparate Impact (DI):} This is defined as the ratio of the probability of unprivileged group vs. privileged group getting a favorable prediction \cite{JMLR:v20:18-262}

\begin{center}
    $DI = P[\hat{S} = 1| Y = 0]/ P[\hat{S} = 1| Y = 1]$
\end{center}

\noindent We convert Disparate Impact (DI) to log scale to improve readability compared with other metrics. \\

\noindent \textit{Statistical Parity Difference (SPD):} This is defined similar to DI but uses the difference between the probabilities. \cite{Calders2010ThreeNB}.

\begin{center}
    $SPD = P[\hat{S} = 1| Y = 0] - P[\hat{S} = 1| Y = 1]$
\end{center}

\noindent \textbf{Individual fairness metrics:}

\noindent \textit{Theil Index (TI):} It measures both the group and individual fairness \cite{10.1145/3219819.3220046}. It is defined using the following equation:
\begin{center}

$TI = \sum_{i=1}^{n} \frac{b_i}{\Bar{a}} ln \frac{b_i}{\Bar{a}}$, where $b_i = \hat{s}_i - s_i + 1$.
\end{center}

\subsection{Experiment Design \& Setup}
\label{subsec:experiment-design-setup}

Each ensemble model has specific requirements for training (e.g., \textit{XGBoost} can handle Null values, but \textit{Random Forest} cannot \textit{etc.}) that we need to handle before we can evaluate them. We used the same preprocessing steps across all the kernels and datasets to ensure consistent comparison. Next, we evaluated the accuracy and fairness of base learners and the final ensemble level and analyzed the results.

For our data preprocessing, we start off by converting all non-numerical features to categorical data i.e. \textit{Binary} or \textit{Ordinal} (e.g., $\textit{{male}: 1}, \textit{{female}: 0}$ or  non-binary levels, like \textit{Marital-Status} to \textit{{Divorced: 0}, {Married: 1}, {Single: 2}} etc.). Next, we remove missing values from our datasets and convert continuous sensitive attributes to categorical (e.g., age $>25$: 1, age$<25$ :0 corresponding to old and young, respectively). These preprocessing steps are necessary for most ensembles and the AIF360 toolkit. We denote the privileged and unprivileged groups and the favorable label for each dataset separately. For example, in \textit{Titanic} dataset, \textit{male} is the unprivileged group and the favorable label is {\textit{Survivied: 1}} i.e. the individual survived the titanic crash. The groups and the labels have been chosen as seen before in literature \cite{martinez2019fairness, 10.1145/3368089.3409704}. Finally, the dataset is shuffled and split into train and test sets using a $70\%-30\%$ split.
For each dataset, we have selected the top 6 kernels by upvotes. We run the preprocessing steps discussed before training the model to evaluate based on accuracy and fairness metrics. We use five fairness metrics and two accuracy measures to generate results for each model. These experiments are repeated ten times, and the mean is reported \cite{10.1145/3287560.3287589}. We used the \textit{IBM AIF 360 Fairness Toolkit} to calculate the fairness metrics.
Finally, a non-zero value for fairness metrics suggests a bias in the model. A positive value of a fairness metric suggests the model is biased against the privileged group and vice-versa.

\section{Motivation and Background}
\label{sec:motivation}

In this section, we use a motivating example to illustrate
the complexity of fairness composition in ensembles and the need to study bias induced by certain ensemble parameters.

\subsection{Motivating example}

Ensemble models are widely deployed to win competitions in online communities like Kaggle due to their superior performances \cite{10.5555/2627435.2697065, inproceedings-OnlineLearning, dietterich2000experimental, bauer1999empirical, HOSNI201989, DBLP:journals/corr/abs-2006-08334}. In prior SE works \cite{10.1145/3368089.3409704,10.1145/3468264.3468536} on fairness, more than 80\% of the models were ensemble based. However, those works did not consider fairness composition of individual learners, its effect on fairness of ensembles and the inherent bias in ensemble methods, which is non-trivial. Hence, not studying fairness composition in ensembles fails to capture the complete fairness of an ML pipeline. Consider the code snippet below of a top-performing model (Voting ensemble) from Kaggle, which is used to predict the income of an individual (\textit{German Credit} dataset).\\

\definecolor{codegreen}{rgb}{0,0.6,0}
\definecolor{codegray}{rgb}{0.5,0.5,0.5}
\definecolor{codepurple}{rgb}{0.58,0,0.82}
\definecolor{backcolour}{HTML}{fffff0}

\lstdefinestyle{mystyle}{
    backgroundcolor=\color{backcolour},
    commentstyle=\color{codegreen},
    keywordstyle=\bfseries\color{magenta},
    numberstyle=\tiny\color{codegray},
    stringstyle=\color{codegreen},

    basicstyle=\fontsize{6}{9}\selectfont,
    breakatwhitespace=false,
    breaklines=true,
    captionpos=b,
    keepspaces=true,
    numbers=left,
    numbersep=5pt,
    showspaces=false,
    xleftmargin=1.5em,
    xrightmargin = 0.5em,
    rulecolor=\color{lightgray},
    showstringspaces=false,
    showtabs=false,
}
\lstset{style=mystyle}
\begin{lstlisting}[language=Python, frame=single]
models = []
models.append(('LGR', LogisticRegression()))
models.append(('LDA', LinearDiscriminantAnalysis()))
models.append(('KNN', KNeighborsClassifier()))
models.append(('CART', DecisionTreeClassifier()))
models.append(('NB', GaussianNB()))
models.append(('RF', RandomForestClassifier()))
models.append(('SVM', SVC(gamma='auto'))
models.append(('XGBM', XGBClassifier()))
models.append(('LGBM', LGBMClassifier()))
model = VotingClassifier(estimators=models, voting='soft')
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

\end{lstlisting}

Voting ensemble is a type of {\em heterogeneous} ensemble that combines the predictions of dissimilar learners. This ensemble comprises multiple learners (lines 2-10) and uses a voting mechanism (line 11) to make the prediction. In \textit{soft} voting, class label (1 or 0) with the higher average probability from the learners is chosen as the final prediction. We found that this ensemble is biased towards female applicants (Protected attribute: \textit{Sex}) in terms of statistical parity difference (SPD:-0.203). In this example, before training the ensemble, a developer must decide the number of learners, select which learners to use, and the voting type (\textit{soft/hard}). However, we found that ML libraries do not provide any fairness recommendations to build ensembles. Do these learners introduce unfairness in the predictions? How does the number of learners impact the fairness of the ensemble? More importantly, we observed that individual learners have their own fairness measures when analyzed in isolation but might result in an unfair model when used in an ensemble. For instance, our analysis shows that dropping XGBClassifier and LGBMClassifier (lines 9-10) can improve fairness by 27\% (SPD:-0.148). Interestingly, we discuss later how these two learners are inherently fair themselves and not responsible for the unfairness.

Furthermore, prior research has shown the impact of hyperparameters on fairness \cite{10.1145/3510003.3510202, 10.1145/3468264.3468536, 10.1145/3368089.3409697}. Ensemble hyperparameters dictate how ensembles combine learners for final prediction. In this example, if developers used \textit{"hard"} voting (line 11), the fairness of the ensemble would improve (SPD: -0.195). Similarly, some of these hyperparameters also affect the design properties of the learners which impacts fairness. \textit{XGBoost} (line 9) is another example of ensemble (boosting). Boosting builds an ensemble of trees (learners) using various methods. What properties of these trees (e.g. tree depth, number of features etc.) and the learning method impact the overall fairness of a boosting ensemble? Exploring these parameters will help developers understand how to design fair ensembles. Therefore, in addition to understanding fairness composition in the learners, it is equally important to understand how the design of ensembles using these parameters impacts fairness.

\subsection{Ensemble learning in ML software}
\label{subsec:ensemble-in-ml}
{\em Ensemble models} are a class of ML classifiers where the predictions from different learners (models) are pooled using a combination method (voting, average, random, etc.) to make the final predictions. In the motivational example above, we only discussed a single type of ensemble. Categories of ensembles are based on homogeneity, learning technique, and ensemble types. All the ensemble types covered in our study and the corresponding classifiers are given in Table \ref{tab:ensemble-types}. There are mainly two categories of ensembles: Sequential and Parallel.

\textbf{\textit{Sequential Ensembles.}} These ensembles sequentially generate base learners. Each learner in the ensemble depends on the previous learners in the sequence because the next learner attempts to correct the wrong predictions from the previous learner and so on \cite{8360836}. AdaBoost is an example of a sequential model where it reweighs (higher) misclassified examples.

\textbf{\textit{Parallel Ensembles.}} Parallel ensembles train individual base learners in parallel and independent of each other. These learner are combined using techniques such as bagging (random sample of data with replacement) or voting, which encourages improved variance \cite{8360836} e.g., \textit{Random Forest.}

\textit{\textbf{Homogeneity of ensembles.}}
Ensemble methods that use single-type base learners are called \textit{homogeneous} models \cite{Elish2013EmpiricalSO}. These individual learners are combined to generate the final result, e.g., {\em XGBoost} and {\em AdaBoost} use decision trees. By contrast, \textit{heterogeneous} ensembles combine the predictions of disimilar individual learners \cite{Elish2013EmpiricalSO}. A popular heterogeneous ensemble method is {\em Voting}. Finally, ensemble method types are divided into {\em Boosting, Bagging, Voting}, \& {\em Stacking} \cite{scikit-learn}.

\begin{table*}[!htbp]
  \centering
  \scriptsize

  \renewcommand{\tabcolsep}{1pt}

  \caption{Ensemble types used in our experiments}
   \begin{tabular}{c|c|m{6.5cm}|c|c}
    \hline
    \multicolumn{1}{c|}{\cellcolor[HTML]{C0C0C0}\textbf{Categories}} & \multicolumn{1}{c|}{\cellcolor[HTML]{C0C0C0}\textbf{Ensemble Types}} & \multicolumn{1}{c|}{\cellcolor[HTML]{C0C0C0}\textbf{Algorithms}} & \multicolumn{1}{c|}{\cellcolor[HTML]{C0C0C0}\textbf{Composition}} & \multicolumn{1}{c}{\cellcolor[HTML]{C0C0C0}\textbf{Classifiers}} \\
    \hline
    \multirow{3}{*}{Sequential} & \multirow{3}{*}{Boosting} & \multirow{3}{*}{\shortstack {Construct n homogeneous estimators sequentially that
    improve \\ predictions based on the previous estimator's
    incorrect \\ predictions}} & Homogeneous & XGBoost \\
          &       &       & Homogeneous & AdaBoost \\
          &       &       & Homogeneous & Gradient Boosting \\
    \hline
    \multirow{5}{*}{Parallel} & \multirow{3}{*}{Bagging} & \multirow{3}{*}{\shortstack{Construct \textit{n} parallel homogeneous models that are aggregated \\ using averaging}} & Homogeneous & Random Forest \\
         &       &       & Homogeneous & ExtraTrees \\
         &       &       & Homogeneous & Bagging Classifier \\
\cline{2-5}          & Voting & Construct a list of n hetereogeneous user-specified weighted classifiers that are aggregated using majority voting or argmax & Heterogeneous & Voting Classifier \\
\cline{2-5}          & Stacking & Construct a list of n hetereogeneous classifiers as base learners and a meta-classifier to decide weights for each learner & Heterogeneous & Stacking Classifier \\
    \hline
    \end{tabular}%
  \label{tab:ensemble-types}%
\end{table*}%

\begin{table}[htbp]
  \centering
  \scriptsize
  \renewcommand{\tabcolsep}{1pt}
  \caption{Summary of the datasets and the number of models collected for each in the benchmark }
    \begin{tabular}{c|c|c|c|c|c|c|c|c|c|c}
    \hline
    \cellcolor[HTML]{C0C0C0}\textbf{Datasets} & \cellcolor[HTML]{C0C0C0}\textbf{PA} &
    \cellcolor[HTML]{C0C0C0}\textbf{Size} &
    \cellcolor[HTML]{C0C0C0}\textbf{\# XGB} & \cellcolor[HTML]{C0C0C0}\textbf{\# ADB} & \cellcolor[HTML]{C0C0C0}\textbf{\# GBC} & \cellcolor[HTML]{C0C0C0}\textbf{\# RF} & \cellcolor[HTML]{C0C0C0}\textbf{\# ET} & \cellcolor[HTML]{C0C0C0}\textbf{\# STK} & \cellcolor[HTML]{C0C0C0}\textbf{\# VT} & \cellcolor[HTML]{C0C0C0}\textbf{Total} \\
    \hline
    \cellcolor[HTML]{EDEDED}\textbf{Adult Census} & \cellcolor[HTML]{EDEDED} sex &
    \cellcolor[HTML]{EDEDED} 32561 &
    \cellcolor[HTML]{EDEDED} 6     & \cellcolor[HTML]{EDEDED} 6     &  \cellcolor[HTML]{EDEDED} 6     & \cellcolor[HTML]{EDEDED} 6     &  \cellcolor[HTML]{EDEDED} 6     &  \cellcolor[HTML]{EDEDED} 1     & \cellcolor[HTML]{EDEDED} 2     & \cellcolor[HTML]{EDEDED} 33 \\
    \textbf{Titanic ML} & sex
    & 891 & 6     & 6     & 6     & 6     & 6     & 6     & 6     & 42 \\
    \cellcolor[HTML]{EDEDED}\textbf{Bank Marketing} & \cellcolor[HTML]{EDEDED} age   &
    \cellcolor[HTML]{EDEDED} 41118 &
    \cellcolor[HTML]{EDEDED} 6     & \cellcolor[HTML]{EDEDED} 6     & \cellcolor[HTML]{EDEDED} 6     & \cellcolor[HTML]{EDEDED} 6     & \cellcolor[HTML]{EDEDED} 2     &  \cellcolor[HTML]{EDEDED} 1     & \cellcolor[HTML]{EDEDED} 2     & \cellcolor[HTML]{EDEDED} 29 \\
    \textbf{German Credit} & sex   & 1000 & 6     & 1     & 1     & 6     & 1     & 1     & 1     & 17 \\
    \hline
    \end{tabular}%
  \label{tab:datasetsummary}%
  \begin{tablenotes}
      \small
      \centering
      \item  XGB: XGBoost, ADB: AdaBoost, GBC: Gradient Boosting
      , RF: Random Forest, ET: Extra Trees, STK: Stacking, VT: Voting
    \end{tablenotes}
\end{table}%

\section{Related Work}
\label{sec:related}

\paragraph{\textbf{Fairness in ML classification}}The ML community has proposed multiple methods to measure \cite{10.1145/2090236.2090255,three-bayes, Chouldechova2017FairPW,10.1145/3278721.3278729, pmlr-v28-zemel13,NIPS2016_9d268236} and mitigate unfairness in ML models \cite{NIPS2016_dc4c44f6, NIPS2017_b8b9c74a, 10.1145/3278721.3278779,three-bayes, Chouldechova2017FairPW}. However, most of these works have focused on the theoretical evaluation of fairness. Recently, the SE community has increasingly shown interest in fairness in ML software \cite{fairnessinsoftware}. Empirical studies have investigated the characteristics of biased models, unfairness in ML pipelines, compared mitigation strategies and developer concerns about fairness \cite{10.1145/3368089.3409704,10.1145/3468264.3468536, 10.1145/3287560.3287589,10.1145/3290605.3300830}. Some research in SE has focused on fairness testing to generate test data inputs and uncover fairness violations \cite{9793943,10.1145/3106237.3106277, 10.1145/3338906.3338937, Fairtest}. Finally, a body of work has identified unfairness in data and proposed appropriate mitigation techniques \cite{10.1145/3468264.3468537, 10.1145/3368089.3409697, DBLP:journals/corr/abs-2111-02038, 9794106}.

\paragraph{\textbf{Ensemble Fairness}}
Grgic-Hlaca \MakeLowercase{\textit{et al.}} \cite{Nina} investigated the impact of fairness in random-selection based ensemble. They showed theoretically that its fairness at the ensemble level is always fairer than its components. Wang \MakeLowercase{\textit{et al.}} \cite{49284} studied the composition of fairness in multi-component recommender systems and presented conditions under which individual components compose fairness. AdaFair \cite{8995403} proposed a fairness-aware AdaBoost model where unfairly classified instances were up-weighted. A recent work \cite{9462068} analyzed and compared seven ML models to show that ensembles were fairer than individual classifiers. Feffer \MakeLowercase{\textit{et al.}} \cite{IBMLale} conducted an empirical study to analyze modular ensembles. They developed a library to find the best configuration using any combination of ensembles and mitigators. In Fair Pipelines \cite{article}, the authors explored the propagation of fairness in multi-stage pipelines where a set of decisions impacts the final result, e.g., the hiring process. MAAT \cite{MAAT} proposes an ensemble approach to improve fairness performance by separately combining models optimized for fairness and accuracy. Finally, Tizpaz-Niari \MakeLowercase{\textit{et al.}} studied the parameter space of ML algorithms and its impact on fairness \cite{10.1145/3510003.3510202}. This work is the closest to our study; however, it proposed a testing approach to tune the parameters for achieving fairness and did not consider ensembles (except \textit{random forest}). Our work has focused on comprehensively evaluating fairness composition in all popular ensemble models and how the different algorithmic design configurations (parameters) impact fairness.

\section{Fair Ensembles Design}
\label{sec:fair-design}

\begin{table*}[]
  \scriptsize
  \centering
  \caption{Ensemble-related hyper-parameters (HP) that can affect the design of fair models}
  \setlength{\tabcolsep}{1pt}
  \setlength\extrarowheight{1pt}
    \begin{tabular}{|l|l|c|c|c|c|c|c|c|c|}
    \hline
    \textbf{HP} & \textbf{Values} & \textbf{Default} & \textbf{ADB} & \textbf{GBC} & \textbf{XGB} & \textbf{RF} & \textbf{ET} & \textbf{Voting} & \textbf{Stacking} \\

    \hline
    n\_estimators & Total number of trees/boosting rounds & 100 & \textcolor[rgb]{ 0,  .69,  .314}{\checkmark} & \textcolor[rgb]{ 0,  .69,  .314}{\checkmark} & \textcolor[rgb]{ 0,  .69,  .314}{\checkmark} &  & \textcolor[rgb]{ 0,  .69,  .314}{\checkmark} & \textcolor[rgb]{ 0,  .69,  .314}{\checkmark} &  \\
    \hline
    booster  & Booster type: {gbtree, gblinear, dart}& gbtree &              & \textcolor[rgb]{ 0,  .69,  .314}{\checkmark} &    &       &       &       &  \\
    \hline
    base\_estimator & \textcolor[rgb]{ .129,  .145,  .161}{Base estimator to fit on random subsets of data} & DecisionTree & \textcolor[rgb]{ 0,  .69,  .314}{\checkmark} &       &        &       &       &    &    \\
    \hline

    bootstrap & Data sampling with replacement & RF: True, ET: False &       &       &       & \textcolor[rgb]{ 0,  .69,  .314}{\checkmark} & \textcolor[rgb]{ 0,  .69,  .314}{\checkmark} &       &  \\
    \hline
    voting & Voting Type: {soft, hard} & hard &       &       &             &       &       & \textcolor[rgb]{ 0,  .69,  .314}{\checkmark} &  \\
    \hline
    estimators & List of individual estimators &  none &    &       &              &       &       & \textcolor[rgb]{ 0,  .69,  .314}{\checkmark} & \textcolor[rgb]{ 0,  .69,  .314}{\checkmark} \\
    \hline
    final\_estimator & Meta-learner to combine learner predictions & Logisitic Regression &       &       &              &       &       &       & \textcolor[rgb]{ 0,  .69,  .314}{\checkmark} \\
    \hline
    \end{tabular}%
    \vspace{-4mm}
  \label{Parameters}%
\end{table*}%

In \S \ref{sec:fairness-in-ensembles}, we found that base learners of ensembles propagate bias. Many bias mitigation techniques applied during model training (\textit{inprocess}) have been successful \cite{10.1145/3368089.3409697,10.1145/3510003.3510202, 10.1145/3278721.3278779}. Techniques applied during the model training phase can assist developers in improving the fairness of ML software. These works have also established the role of hyperparameters in mitigating and amplifying fairness bugs (unfairness) in ML software. If we understand and identify what ensemble parameters and design choices affect the fairness, we can mitigate inherent bias in ensembles. Moreover, it will help developers, and libraries better explain fairness bugs in the ensemble hyperparameter space. This section explores the hyperparameter design space for ensembles to boost fairness performance. We have found that some hyperparameters directly affect the fairness of ensembles. Specifically, we evaluate how ensembles can be designed to be fair using ensemble hyperparameters summarized in Table \ref{Parameters}. We use the Scott-Knott test to determine the significance of our results. Our findings provide a comprehensive review of all ensemble hyperparameters.

\stepcounter{finding}
\begin{mdframed}[style = exampledefault]
\noindent \textbf{Finding \thefinding:} Developers should carefully choose dropout regularization to balance fairness and overfitting.
\end{mdframed}
\vspace{-5pt}
Our analysis shows that dropout impacts fairness in relation to the underlying data properties. Vinayak and Gilad-Bachrach \cite{rashmi2015dart} proposed DART, a dropout technique derived from deep neural networks, for boosted trees. An ensemble of boosted regression trees suffers from over-specification, i.e., the trees added at the end have little contribution to the final result \cite{rashmi2015dart}. \textit{DART} alleviates this by constructing the next tree from the residuals of a random sample of the previous trees. In \textit{XGBoost}, the \textit{rate-drop} ([0-1]) parameter controls this sampling rate. No trees are dropped on the lower end of this rate, while on the higher extreme, all trees are dropped. We investigate the efficacy of DART with \textit{ratedrop = 0.5}, in reducing unfairness in boosting models by comparing it with the default \textit{XGBoost} booster \textit{gbtree}. We analyze the change in performance and fairness of dropout in Figure \ref{Dart:comparison}.

\begin{figure}[]
  \centering
    \includegraphics[width=\linewidth]{Figures/Dart.pdf}
    \caption{Change in accuracy and total bias when using DART. Negative value denotes lower fairness \& accuracy}
    \label{Dart:comparison}
    \vspace{-6mm}
\end{figure}

From figure \ref{Dart:comparison}, we can see that dropout can impact fairness of boosting models. For example, in \textit{Adult} dataset, initial trees exhibit less unfairness than latter. Using dropout, the subsequent trees only learn on a random sample of initial trees, which in this case are fairer. This improves the fairness of the models. The opposite is observed in \textit{Titanic} dataset. In both scenarios, the change in accuracy is less than 0.1, but a significant impact is seen on fairness. Therefore, developers should be cautious about the effect of regularization on fairness. More research is needed to understand fairness-overfitting trade-off, and develop fair regularization methods.

\stepcounter{finding}
\begin{mdframed}[style = exampledefault]
\noindent \textbf{Finding \thefinding:} Randomness in feature splitting does not improve fairness in bagging models.
\end{mdframed}
\vspace{-5pt}

\textit{Random Forest} improves the variance of the model by introducing randomness to the process of model building by randomly selecting features. \textit{Extra Trees} introduces additional randomness by randomly finding the splits for each feature and then selecting the best split from them, i.e., independent of the target variable. In contrast, \textit{Random Forest} finds the best split for each feature which has been shown to improve accuracy \cite{geurts2006extremely}. However, no work has studied its effect on fairness. Here, we ask whether randomness at the feature splitting level causes bagging models to be unfair.

To investigate this, we compare \textit{Random Forest} and \textit{Extra Tree} models in our benchmark. We keep the rest of the parameters and data split the same. Each model is run ten times, and the mean is reported in Table \ref{tab:RFvsET}. For all datasets except \textit{German Credit}, the test showed that \textit{Extra Tree} models with random splits were more biased compared to optimal splits (\textit{RF}). This is a key finding because this suggests that a split chosen independently of the target is still more unfair than an optimal split. However, in the fairer dataset (\textit{German} Credit), we observe no difference in fairness. Regarding bias mitigation methods, our results suggest randomness in feature split-point might not be an effective way to tackle bias in decision tree-based models.

\begin{table}[htbp]
  \scriptsize
  \centering

  \caption{Mean total fairness in Random Forest (RF) and ExtraTrees (ET) models. $^*$ denotes the top rank based on Scott-Knott significance rank test for \textit{each} dataset.}
  \resizebox{\columnwidth}{!}{
    \begin{tabular}{ccccccccc}
    \toprule
          & \multicolumn{2}{c}{\textbf{Titanic}} & \multicolumn{2}{c}{\textbf{Adult}} & \multicolumn{2}{c}{\textbf{Bank Marketing}} & \multicolumn{2}{c}{\textbf{German Credit}} \\
\cmidrule{1-9}          & RF$^{*}$    & ET    & RF$^{*}$    & ET    & RF$^{*}$    & ET    & RF    & ET \\
    \toprule
    Model1 & -2.04 & -1.96 & -1.33 & -1.32 & 1.16
 & 1.35  & 0.03  & 0.02 \\
    Model2 & -4.43 & -5.37 & -1.29 & -1.30 & 1.37
  & 1.41  & 0.43  & 0.29 \\
    Model3 & -3.38 & -4.13 & -1.32 & -1.45 & 0.96  & 1.07  & 0.02  & -0.04 \\
    Model4 & -2.42 & -2.46 & -1.21 & -1.29 & 1.36  & 1.35  & -0.16 & -0.09 \\
    Model5 & -2.98 & -3.38 & -1.48 & -1.51 & 1.61
  & 1.51
  & 0.00  & -0.04 \\
    Model6 & -2.61 & -2.61 & -1.47 & -1.41 & 0.98  & 1.26  & -0.04 & -0.28 \\
    \bottomrule
    \end{tabular}}%

  \label{tab:RFvsET}%
\end{table}%

\stepcounter{finding}
\begin{mdframed}[style = exampledefault]
\noindent \textbf{Finding \thefinding:} The uncertainty in
classifiers can have a large impact on fairness in voting classifiers.
\end{mdframed}
\vspace{-5pt}

A \textit{Voting} classifier is an ensemble method where the prediction is based on the probabilities of each base learner within the ensemble. Voting classifiers are of two types, \textit{Soft} and \textit{Hard} Voting. In hard Voting, the label with the majority of votes from the base learners is the final prediction, whereas, in soft voting, it is based on the average of the probabilities of each output class. If the average probability of a class is less than 0.5, class 0 is predicted and vice-versa.
We investigate the effect of the voting type on fairness and found that the uncertainty in the model prediction can have a large impact on fairness. For instance, \textit{AC-VT5} uses soft voting with \textit{Logistic Regression (LR), Random Forest (RF), KNN}, and \textit{Decision Tree (DT)} as base classifiers. As shown in Table \ref{tab:voting}, \textit{DT} introduces significant unfairness when used in soft voting compared to hard. We found that \textit{DT} has an output class probability of either 1 or 0 while other classifiers are in the range [0,1]. Figure \ref{Voting} shows the output probabilities for AC-VT5. We observe that DT has higher extreme probabilities compared to others. In this case, the average is skewed by the extreme probabilities of \textit{DT}. This changes the prediction for 558 out of 9049 test samples. And since \textit{DT} is less fair than other classifiers, the overall unfairness also increases. For hard voting, equal weight is given to each classifier. In that case, the other three classifiers, which are fairer, won the majority vote. For some models, soft voting was fairer than hard, e.g., \textit{AC-VT3}, which shows that base learners' uncertainties can impact fairness in both voting types. This suggests the need to develop frameworks to measure model uncertainties and their fairness at a component level to aid developers in designing fair voting ensembles. Our analysis should also encourage further research in fairness-aware weighting techniques to handle fairness issues arising from model uncertainties.

\begin{figure}[b]
  \centering
    \includegraphics[width=\columnwidth]{Figures/voting.pdf}
    \caption{Frequency of output class probabilities for base learners in AC-VT5}
    \label{Voting}
    \vspace{-5mm}
\end{figure}

\begin{table}[]
  \centering
  \scriptsize
  \renewcommand{\tabcolsep}{1pt}
  \caption{Soft vs Hard Voting in \textit{AC-VT5}}
    \begin{tabular}{|c||c|c|c|c|c||c|c|c|c|c|}
    \hhline{-||----------}
    \multicolumn{1}{|c||}{} & \multicolumn{5}{c| |}{\cellcolor[HTML]{EFEFEF}\textbf{Soft}}     & \multicolumn{5}{c|}{\cellcolor[HTML]{EFEFEF}\textbf{Hard}} \\
    \hhline{-||-----||-----}
    \cellcolor[HTML]{EFEFEF} & \cellcolor[rgb]{ 1,  1,  1}\textbf{Voting} & \cellcolor[rgb]{ 1,  1,  1}\textbf{LR} & \cellcolor[rgb]{ 1,  1,  1}\textbf{RF} & \cellcolor[rgb]{ 1,  1,  1}\textbf{KNN} & \cellcolor[rgb]{ 1,  1,  1}\textbf{DT} & \cellcolor[rgb]{ 1,  1,  1}\textbf{Voting} & \cellcolor[rgb]{ 1,  1,  1}\textbf{LR} & \cellcolor[rgb]{ 1,  1,  1}\textbf{RF} & \cellcolor[rgb]{ 1,  1,  1}\textbf{KNN} & {\cellcolor[rgb]{ 1,  1,  1}\textbf{DT}}\\
    \hhline{-||-----||-----}
    \cellcolor[HTML]{EFEFEF}\textbf{Acc} & \cellcolor[rgb]{ 1,  1,  1}   0.83 & \cellcolor[rgb]{ 1,  1,  1}   0.78 & \cellcolor[rgb]{ 1,  1,  1}   0.81 & \cellcolor[rgb]{ 1,  1,  1}   0.81 & \cellcolor[rgb]{ 1,  1,  1}0.83 & \cellcolor[rgb]{ 1,  1,  1}0.81 & \cellcolor[rgb]{ 1,  1,  1}0.78 & \cellcolor[rgb]{ 1,  1,  1}0.81 & \cellcolor[rgb]{ 1,  1,  1}0.81 & \cellcolor[rgb]{ 1,  1,  1}0.81 \\
    \hhline{-||-----||-----}
    \cellcolor[HTML]{EFEFEF} \textbf{F1} & \cellcolor[rgb]{ 1,  1,  1}0.56 & \cellcolor[rgb]{ 1,  1,  1}0.41 & \cellcolor[rgb]{ 1,  1,  1}0.44 & \cellcolor[rgb]{ 1,  1,  1}0.43 & \cellcolor[rgb]{ 1,  1,  1}0.56 & \cellcolor[rgb]{ 1,  1,  1}0.44 & \cellcolor[rgb]{ 1,  1,  1}0.41 & \cellcolor[rgb]{ 1,  1,  1}0.41 & \cellcolor[rgb]{ 1,  1,  1}0.45 & \cellcolor[rgb]{ 1,  1,  1}0.44 \\
    \hhline{-||-----||-----}
    \cellcolor[HTML]{EFEFEF} \textbf{SPD} & \cellcolor[rgb]{ 1,  1,  1}-0.14 & \cellcolor[rgb]{ 1,  1,  1}-0.06 & \cellcolor[rgb]{ 1,  1,  1}-0.08 & \cellcolor[rgb]{ 1,  1,  1}-0.07 & \cellcolor[HTML]{FFCCC9}-0.14 & \cellcolor[rgb]{ 1,  1,  1}-0.07 & \cellcolor[rgb]{ 1,  1,  1}-0.06 & \cellcolor[rgb]{ 1,  1,  1}-0.06 & \cellcolor[rgb]{ 1,  1,  1}-0.08 & \cellcolor[rgb]{ .882,  1,  .906}-0.07 \\
    \hhline{-||-----||-----}
    \cellcolor[HTML]{EFEFEF} \textbf{EOD} & \cellcolor[rgb]{ 1,  1,  1}-0.14 & \cellcolor[rgb]{ 1,  1,  1}0 & \cellcolor[rgb]{ 1,  1,  1}0 & \cellcolor[rgb]{ 1,  1,  1}0 & \cellcolor[HTML]{FFCCC9}-0.14 & \cellcolor[rgb]{ 1,  1,  1}-0.03 & \cellcolor[rgb]{ 1,  1,  1}0 & \cellcolor[rgb]{ 1,  1,  1}-0.03 & \cellcolor[rgb]{ 1,  1,  1}-0.04 & \cellcolor[rgb]{ .882,  1,  .906}-0.03 \\
    \hhline{-||-----||-----}
    \cellcolor[HTML]{EFEFEF} \textbf{AOD} & \cellcolor[rgb]{ 1,  1,  1}0.1 & \cellcolor[rgb]{ 1,  1,  1}0.01 & \cellcolor[rgb]{ 1,  1,  1}0.02 & \cellcolor[rgb]{ 1,  1,  1}0.02 & \cellcolor[HTML]{FFCCC9}0.1 & \cellcolor[rgb]{ 1,  1,  1}0.02 & \cellcolor[rgb]{ 1,  1,  1}0.01 & \cellcolor[rgb]{ 1,  1,  1}0.02 & \cellcolor[rgb]{ 1,  1,  1}0.03 & \cellcolor[rgb]{ .882,  1,  .906}0.02 \\
    \hhline{-||-----||-----}
    \cellcolor[HTML]{EFEFEF} \textbf{DI} & \cellcolor[rgb]{ 1,  1,  1}-1.43 & \cellcolor[rgb]{ 1,  1,  1}-0.64 & \cellcolor[rgb]{ 1,  1,  1}-1.13 & \cellcolor[rgb]{ 1,  1,  1}-1.02 & \cellcolor[HTML]{FFCCC9}-1.43 & \cellcolor[rgb]{ 1,  1,  1}-1.08 & \cellcolor[rgb]{ 1,  1,  1}-0.64 & \cellcolor[rgb]{ 1,  1,  1}-1.11 & \cellcolor[rgb]{ 1,  1,  1}-1.02 & \cellcolor[rgb]{ .882,  1,  .906}-1.08 \\
    \hhline{-||-----||-----}
    \cellcolor[HTML]{EFEFEF} \textbf{TI} & \cellcolor[rgb]{ 1,  1,  1}0.17 & \cellcolor[rgb]{ 1,  1,  1}0.21 & \cellcolor[rgb]{ 1,  1,  1}0.2 & \cellcolor[rgb]{ 1,  1,  1}0.2 & \cellcolor[rgb]{ .882,  1,  .906}0.17 & \cellcolor[rgb]{ 1,  1,  1}0.2 & \cellcolor[rgb]{ 1,  1,  1}0.22 & \cellcolor[rgb]{ 1,  1,  1}0.21 & \cellcolor[rgb]{ 1,  1,  1}0.2 & \cellcolor[HTML]{FFCCC9}0.2 \\
    \hhline{-||-----||-----}
    \end{tabular}%
  \label{tab:voting}%
  \vspace{-4mm}
\end{table}%

\stepcounter{finding}
\begin{mdframed}[style = exampledefault]
\noindent \textbf{Finding \thefinding:} Two-layer stacking can significantly reduce unfairness.
\end{mdframed}
\vspace{-5pt}

All of the \textit{Titanic} ML stacking models shown in Figure \ref{All_models} exhibit similar bias except \textit{TM-STK5}, which is the least biased model for all fairness metrics except Thiel Index (TI). On closer inspection, we found out that \textit{TM-STK5} uses a two-layered stacking approach where a second layer of base learners act as the meta-learner, which causes the model to be fairer. The model construct is shown below:

\definecolor{codegreen}{rgb}{0,0.6,0}
\definecolor{codegray}{rgb}{0.5,0.5,0.5}
\definecolor{codepurple}{rgb}{0.58,0,0.82}
\definecolor{backcolour}{HTML}{fffff0}

\lstdefinestyle{mystyle}{
    backgroundcolor=\color{backcolour},
    commentstyle=\color{codegreen},
    keywordstyle=\bfseries\color{magenta},
    numberstyle=\tiny\color{codegray},
    stringstyle=\color{codegreen},

    basicstyle=\fontsize{6}{9}\selectfont,
    breakatwhitespace=false,
    breaklines=true,
    captionpos=b,
    keepspaces=true,
    numbers=left,
    numbersep=5pt,
    showspaces=false,
    xleftmargin=1.2em,
    xrightmargin = 0.5em,
    rulecolor=\color{lightgray},
    showstringspaces=false,
    showtabs=false,
}
\lstset{style=mystyle, frame = single}

\begin{lstlisting}[language=Python]
layer_one_estimators = [('rf_1', RandomForestClassifier(n_estimators=40, random_state=42)),('knn_1', KNeighborsClassifier(n_neighbors=6))]
layer_two_estimators = [('rf_2', RandomForestClassifier(n_estimators=40, random_state=42)),('xg_2', XGBClassifier(objective = 'binary:logistic', colsample_bytree = 0.8, learning_rate = 0.3, max_depth = 7, min_child_weight = 3, n_estimators = 100, subsample = 0.6))]
layer_two = StackingClassifier(estimators=layer_two_estimators, final_estimator=XGBClassifier(n_estimators  = 100))
model =  StackingClassifier(estimators=layer_one_estimators,final_estimator=layer_two)}
\end{lstlisting}

We validate our finding by training all stacking models in our benchmark using this two-layered nested stacking approach. To ensure consistency, we did not change the kernel's feature set or any preprocessing method. The results are shown in Figure \ref{Fig:Stacking_Nested}. For all stacking models in our benchmark, every model significantly improved in all fairness measures except \textit{Thiel Index}. Achieving fairness in terms of all fairness metrics is difficult \cite{Chouldechova2017FairPW, https://doi.org/10.48550/arxiv.1703.09207}. Moreover, accuracy measures did not degrade significantly. For example, \textit{TM-STK6} improved \textit{SPD} scores by $28\%$ while accuracy dropping only $4.68\%$. Overall across all datasets, the accuracy score dropped by $6.50\%$ while the \textit{SPD} improved by $31.8\%$.

\begin{figure}[]
  \footnotesize
  \centering

  \includegraphics[]{Figures/merged-stacking.pdf}
    \caption{Performance \& fairness changes after using two-layered stacking (Top) and XGB as meta-learner (Bottom). A positive change indicates performance/fairness increase and vice-versa}
    \label{Fig:Stacking_Nested}
    \vspace{-2em}
\end{figure}

In stacking, the first layer uses a list of base learners to generate a set of predictions and a meta-learner to learn from them. However, instead of using a single classifier as the meta-learner, the predictions are fed into another set of base learners in a two-layered approach. This ensures that the outcome is not based on a single meta-learner. Instead, the second layer of learners creates a new set of predictions, which are then fed into the second layer's final meta-classifier. We compare this approach to simply using an ensemble model (\textit{XGB}) as the meta-learner in default stacking models and found similar patterns in fairness measures (Figure \ref{Fig:Stacking_Nested} bottom). Our results did not significantly vary between using other ensembles as the meta-learner. This supports observations that ensembles are fairer than standalone classifiers. Therefore, developers should use an ensemble as a meta-learner or the two-layer approach to improve the fairness of stacking models.

\section{Threats to Validity}
\label{sec:threats}

\noindent \textbf{Benchmark:} We ensure quality of the benchmark by collecting only high-quality kernels from Kaggle (atleast 5 upvotes). Additionally, we only consider models that are runnable, include the protected attribute in training, and have an accuracy greater than 65\%, similar to \cite{10.1145/3368089.3409704}. Finally, we select the top 6 (upvotes) models for each ensemble type. \\
\noindent \textbf{Sampling Bias:} To the best of our knowledge, this is the most extensive review of popular ensembles. Moreover, conclusions are supported by statistical tests across four datasets. However, they may change slightly if other datasets are used. \\
\textbf{External validity:} To avoid the threat of non-generalized findings, we conduct experiments on four different datasets for each ensemble type and compare across multiple ensemble algorithms for both boosting and bagging. Moreover, we use multiple fairness metrics and verify our results by running the experiment multiple (ten) times and using mean of the values.
\section{Fairness in Ensembles and its composition}
\label{sec:fairness-in-ensembles}

In this section, we explore the state of fairness in ensembles and its composition in all popular ensemble methods.

\subsection{State of fairness in ensemble models}
\label{subsubsec:state-of-fairness}

Before understanding the composition of fairness in ensembles, we first investigate how different ensemble techniques impact fairness \textbf{(RQ1)}. Are certain ensemble classifiers more unfair? Does the architecture of an ensemble method (stacking, boosting, etc.) contribute to fairness? Does any particular ensemble classifier exhibit a better fairness-accuracy trade-off? To answer these questions, we experiment to evaluate the fairness of ensemble models using a diverse set of metrics. Table \ref{Table_Comparison_1} shows the mean fairness for all ensembles. Figure \ref{All_models} illustrates the cumulative fairness for all the 168 models.

We found that ensemble models show similar fairness patterns specific to each dataset and that some ensemble classifiers exhibit more unfairness than others. To investigate whether these differences are significant and directly impacted by ensemble type, we compare their fairness and accuracy using Scott-Knott \cite{7194626} ranking test. The Scott-Knott test compares two distributions using a recursive bi-clustering algorithm and provides a rank; if the difference between them is insignificant, they get the same rank. Otherwise, a higher rank denotes better performance. In our experiment, the Scott-Knott test ranks the classifiers from $1^{st}$ to $5^{th}$ (some classifiers are assigned the same rank) for each metric. Even though each ensemble classifier's unfairness pattern is similar to the dataset, certain ensembles are fairer than others.

\stepcounter{finding}
\begin{mdframed}[style = exampledefault]
\noindent \textbf{Finding \thefinding:} Among all the ensemble models, XGBoost exhibits the best accuracy-fairness trade-off.
\end{mdframed}
\vspace{-10pt}

From Table III, we can see that the fairness of the ensemble
classifiers varies across the datasets. Moreover, we see that
the fairness of the classifiers within an ensemble type also varies. For instance, \textit{AdaBoost} is the most unfair model in the \textit{Adult} dataset, while \textit{XGBoost} is the fairest, both belonging to boosting. However, we observe that for the more biased \textit{Titanic} and \textit{Adult} datasets, \textit{XGBoost} has the top rank in 8 out of 10 fairness metrics with a mean fairness rank of 1.4 and 1.6, respectively. Yet, \textit{XGBoost} is ranked $2^{nd}$ and $1^{st}$ in terms of accuracy, respectively. The average performance change in accuracy and f1 score is less than 0.01. On the contrary, the cumulative mean fairness for \textit{XGBoost} models is $14\%$ more than the next most fair model (\textit{GBC}) in Titanic. For the other datasets, we observe a similar pattern; however, the difference is lesser. Upon further investigation, we found that boosting method and base learner design are responsible for the fairer performance of \textit{XGBoost}. Homogeneous ensembles use decision trees as the base learner, and the construction of these trees differs among them. For example, the depth of the decision tree in \textit{AdaBoost}, \textit{GradientBoosting}, \textit{XGBoost} is one, three and six respectively. Lower tree depth means fewer features are selected, which has been shown to often increase unfairness \cite{9402057, 10.1145/3368089.3409704}.
This is also one of the reasons behind \textit{AdaBoost} being the most unfair model (the lowest rank in 13 out of 16 metrics). Importantly, we found that the fairness of an ensemble is determined by the composition of fairness within these base learners and the learning method (boosting, bagging, etc.). In the next section, we investigate the properties of the base learners to understand how to compose fair ensembles in more detail.

\begin{figure*}[t]
  \centering
    \includegraphics[width=\linewidth]{Figures/High-Level-cropped.pdf}
    \caption{Cumulative bias and performance of all ensembles. The bars represent mean values, and the dots the models}
    \label{All_models}
\end{figure*}

\stepcounter{finding}
\begin{mdframed}[style = exampledefault]
\noindent \textbf{Finding \thefinding:} Fairness measures show more instability compared to accuracy metrics.
\end{mdframed}
\vspace{-10pt}

Prior works have shown that ensembles improve the stability of accuracy metrics by aggregating multiple learners trained on subsets of data (bootstrap sampling) \cite{Bellamy2018AIF3, breiman1996bagging}. Intuitively, we explore the variance exhibited by fairness metrics in ensembles. Interestingly, despite aggregating multiple learners, the stability of fairness measures in ensembles still suffers, especially in smaller datasets. This is attributed to the change in data distribution after random train/test splits in smaller datasets \cite{10.1145/3287560.3287589, 10.1145/3368089.3409704}. For larger datasets (\textit{Adult, Bank Marketing}), the standard deviation for all fairness measures is less than 0.02. For smaller datasets, the average standard deviation of the metrics is shown in Figure \ref{fig:Std}. Firstly, we observe that the stability of fairness metrics remains consistent between all the ensembles for a specific dataset. Furthermore, we observed that group fairness measures exhibit higher variability than individual measures (TI). Surprisingly, heterogeneous models also exhibit instability despite using dissimilar learners to reduce variance. From Figure \ref{Sequential_Ensembles}, we also see that the volatility in fairness metrics is greater than in accuracy metrics for homogeneous models. Given a random train/test split, it might cause the model to seem fairer than it is. Hence, even with improved stability in fairness compared to non-ensemble methods, developers should evaluate the training set and repeat training over multiple runs in ensembles.

\begin{figure}[]
  \centering
    \includegraphics[width=\linewidth]{Figures/Standard_Deviation.pdf}
    \caption{Standard deviation of fairness metrics over multiple experiments. Other datasets have very low standard deviation.}
    \label{fig:Std}
    \vspace{-5mm}
\end{figure}

\stepcounter{finding}
\begin{mdframed}[style = exampledefault]
\noindent \textbf{Finding \thefinding:} Libraries do not provide API support to measure fairness of base learners in ensembles
\end{mdframed}
\vspace{-5pt}

Biswas and Rajan \cite{10.1145/3368089.3409704} discussed that hyperparameter optimization goals induce unfairness. Whereas ensemble models also have parameters that can be tuned, heterogeneous models have a special case where the developer needs to select the individual base learners. Libraries do not provide any recommendations to developers, who try to select a diverse set of learners to improve accuracy. However, this might not always result in a fair ensemble. For example, \textit{BM-STK3} has Statistical Parity Difference (SPD) of 0.13 but if we drop GuassianNB learner, the SPD improves to 0.11 with better accuracy. Heterogeneous \textit{Voting} models uses weighted voting, while \textit{Stacking} uses a meta-learner to select the best configuration of learners for each prediction. Since libraries do not provide API support to measure the fairness of base learners, especially in combinations with other learners at the prediction level, it provides developers with little information on how to weigh and select individual learners to generate the final predictions. Similarly, understanding fairness composition in base learners of homogeneous models can help the developer identify fairness issues such as bias in specific features (e.g., decision trees learners in random forest randomly select features). Therefore, API support to measure fairness in base learners can help developers better understand \& detect unfairness in ensembles.

\begin{figure*}[]
  \centering
    \includegraphics[width=\linewidth]{Figures/Sequential_Ensembles.pdf}
    \caption{Composition of fairness in homogeneous ensembles with respect to base learners. Default no. of trees: T100}
    \label{Sequential_Ensembles}
\end{figure*}

\subsection{How does fairness compose in ensembles?}

\label{5.3}

In this section, we investigate the composition of fairness in ensembles. We postulate that the underlying unfairness of ensembles is a product of the composition of fairness in base learners and the learning method. All homogeneous models use a decision tree as the base learner, whereas heterogeneous models can take any ML model. We investigate how fairness composes in these base learners and how it is propagated by the learning techniques \textbf{(RQ2)}.
In general, our findings show that the complexity of base learners significantly impacts the fairness of ensembles and that more research is needed to develop fair learning techniques in ensembles.

\stepcounter{finding}
\begin{mdframed}[style = exampledefault]
\noindent \textbf{Finding \thefinding:}
The unfairness of homogeneous ensembles is caused by the complexity of the base learner and dataset characteristics.
\end{mdframed}
\vspace{-5pt}

In Figure \ref{Sequential_Ensembles}, we plotted the most biased and the least biased homogeneous ensembles in our benchmark. We see how the fairness of the base learners directly contributes to the fairness of ensembles during training. However, we observe different fairness composition patterns in terms of the datasets. For example, in \textit{Titanic}, the fairness of the boosting ensembles improves while the opposite is observed for some fairness metrics in \textit{German Credit}. The variation in fairness patterns is also seen in specific classifiers. From Figure \ref{All_models}, we observe that all \textit{TM-XGB} models show similar bias except, \textit{TM-XGB4}. We investigate the difference in unfairness by comparing all the parameters of the base learner decision tree with the other \textit{XGB} models and found that \textit{TM-XGB4} uses a shallow decision tree with $max\_depth:2$, which is causing the unfairness to amplify. The model construct is shown below:

\begin{lstlisting}[language=Python]
model = XGBClassifier(n_estimators= 500,max_depth=2, subsample=0.5, learning_rate=0.1)
\end{lstlisting}

\textit{MaxDepth} sets the maximum depth of the decision tree. The depth of the tree is defined as the number of splits (nodes), where the feature to be split is chosen based on the highest information gain among the features. Deeper trees are more complex and reduce errors \cite{10.1023/A:1022643204877}. For \textit{XGBoost} models, the default depth is 6. Our analysis showed that the protected attribute (\textit{Sex}) has the highest information gain among all the features in the \textit{Titanic} dataset. Therefore, the protected attribute is the most important feature to split on at the tree's root, resulting in a high degree of unfairness in \textit{TM-XGB4}.
Base learners in all boosting models use the best feature to split, which improves accuracy. However, it has been shown that unfairness is encoded in specific features \cite{10.5555/3504035.3504042}. If these features are also among the best features of a dataset, a shallower ensemble will be more unfair due to a reduced number of features. This corroborates similar observations in the literature \cite{10.1145/3468264.3468536, 9402057}. We observe the same pattern for all boosting models. For example, \textit{AdaBoost} and \textit{GradientBoosting} exhibit more bias than \textit{XGBoost} because of shallower base learners (1 and 3, respectively). In Figure \ref{All_models}, \textit{TM-GBC1} is fairer because the learner is deeper (depth:5).

Finally, further analysis of the properties of a decision tree suggests that regularization parameters like min samples leaf and max-leaf nodes also impact tree depth, hence affecting the fairness of the ensemble. Therefore, it is important to carefully balance the complexity of the tree-based base learners for homogeneous models with the fairness outcomes, especially with the underlying properties of the data.

\stepcounter{finding}
\begin{mdframed}[style = exampledefault]
\noindent \textbf{Finding \thefinding:} \textit{Gradient-based} composition propagates more unfairness compared to \textit{Adaptive} boosting models.

\end{mdframed}
\vspace{-5pt}
We have established that base learners and the underlying data properties influence the unfairness of homogeneous models. However, boosting models additionally use a learning technique to sequentially improve the predictions of the model. \textit{XGBoost} and \textit{GradientBoosting} models uses Gradient-based optimization and \textit{AdaBoost} uses an adaptive weighting technique. We compare these techniques by training the boosting models on the same base learner decision tree (depth:6). We only use \textit{XGBoost} and choose this depth in our experiment since our analysis (Table \ref{Table_Comparison_1}) showed that it is the most fair boosting model. The results are shown in Figure \ref{GradientvsADB}. For all the models, except GC1, we see that Adaptive learning is fairer than Gradient optimization. We use the Scott-Knott rank test to test statistical significance. Accordingly, we observed that adaptive learning outperformed gradient optimization in all datasets, except German Credit where the difference was not statistically significant. Consequently, we can see that adaptive learning propagates less bias in highly biased datasets. Our analysis should help guide further research into designing fair learning techniques for boosting ensembles.

\begin{figure}[]
  \footnotesize
  \centering
    \includegraphics[width=\linewidth]{Figures/Gradient-vs-Adaptive.pdf}
    \caption{Cumulative fairness in adaptive vs gradient learning}
    \label{GradientvsADB}
    \vspace{-4mm}
\end{figure}