Many high-stake applications such as finance, hiring, admissions, criminal justice use algorithmic decision-making frequently. In some cases, machine learning models make better decisions than human can do [1, 2]. But there are many scenarios where machine learning software has been found to be biased and generating arguably unfair decisions. Google’s sentiment analyzer model which determines positive or negative sentiment, gives negative score to the sentences such as ‘I am a Jew’, and ‘I am homosexual’
. Facial recognition software which predicts characteristics such as gender, age from images has been found to have a much higher error rate for dark-skinned women compared to light-skinned men. A popular photo tagging model has assigned animal category labels to dark skinned people . Recidivism assessment models used by the criminal justice system have been found to be more likely to falsely label black defendants as future criminals at almost twice the rate as white defendants . Amazon.com stopped using automated job recruiting model after detection of bias against women. Cathy O’Neil’ provided even more examples of unfair decisions made by software in her book “Weapons of Math Destruction”. She argued that machine learning software generates models that are full of bias. Hence, this is one of the reasons their application results in unfair decisions.
Machine learning software, by its nature, is always a form of statistical discrimination. The discrimination becomes objectionable when it places certain privileged groups at systematic advantage and certain unprivileged groups at systematic disadvantage. In certain situations, such as employment (hiring and firing), discrimination is not only objectionable, but illegal.
Issues of fairness have been explored in many recent papers in the SE research literature. Angell et al.  commented that issues of fairness are analogous to other measures of software quality. Galhotra and his colleagues discussed how to efficiently generate test cases to test for discrimination. Udeshi et al.  worked on generating discriminatory inputs for machine learning software. Albarghouthi et al.  explored if fairness can be wired into annotations within a program while Tramer et al. proposed different ways to measure discrimination .
All the above SE research detects unfairness. Our work takes a step further and asks how to mitigate unfairness. We propose that every machine learning model must go through fairness testing phase before it is applied. If bias is found, then the model needs to be optimized. Hence, we have converted “discrimination problem” into an optimization problem. We think that if fairness becomes a goal while learning, then the models created in that way will generate fairer results. In this study, we investigated whether model parameter tuning can help us to make the model fair or not.
In machine learning, many hyperparameters control inductive process ; e.g. the ‘splitter’ of CART . They are very important because they directly control the behaviors of the training algorithm and impact the performance of the model. Therefore, the selection of appropriate parameters plays a critical role in the performance of machine learning models. Our study applies hyperparameter optimization to make a model fair without losing predictive power. So, it becomes multiobjective optimization problem as we are dealing with more than one objective.
Ii But is this a Problem for Software Engineers?
We are not the only ones to assert that software fairness is a concern that must be addressed by software engineers. Other SE researchers are also exploring this issues [9, 1, 2]. For example, IEEE/ACM recently organized a workshop on software fairness called Fairware 2018111http://fairware.cs.umass.edu/.
Nevertheless, when discussing this work with colleagues, we are still sometimes asked if this problem should or can be solved by software engineers. We reply that:
It should be the goal of software developers to ensure that software conforms to its required ethical standards.
Even if we think that fairness is not our problem, our users may disagree. When users discover problems with software, it is the job of the person maintaining that software (i.e. a software engineer) to fix that problem.
Further, we also think that this problem can be solved by software engineers. Hyperparameter optimization is now a standard tool in software analytics [15, 16]. What we are arguing here is that now that those same tools, that have been matured within the SE community (by SE researchers and practitioners), can now be applied to other problems (e.g. as discussed in this paper, how to mitigate unfair software).
We say that a label is called favorable label if its value corresponds to an outcome that gives an advantage to the receiver. Examples like - being hired for a job, receiving a loan. Protected attribute is an attribute that divides a population into two groups that have difference in terms of benefit received. Like - sex, race. These attributes are not universal, but are specific to application. Group fairness is the goal that based on the protected attribute, privileged and unprivileged groups will be treated similarly. Individual fairness is the goal of similar individuals will receive similar outcomes. Our paper studies Group fairness only. By definition, “Bias is a systematic error ” . Our main concern is unwanted bias that puts privileged groups at a systematic advantage and unprivileged groups at a systematic disadvantage. A fairness metric is a quantification of unwanted bias in models or training data . We used two such fairness metrics in our experiment-
Equal Opportunity Difference(EOD): Delta in true positive rates in unprivileged and privileged groups .
Both are computed using the input and output datasets to a classifier. A value of 0 implies that both groups have equal benefit, a value lesser than 0 implies higher benefit for the privileged group and a value greater than 0 implies higher benefit for the unprivileged group. In this study, we have taken absolute value of these metrics.
Iv-a Hyperparameter Optimization
Hyperparameter optimization is the process of searching the most optimal hyperparameters in machine learning learners  . There are four common algorithms: grid search, random search, Bayesian optimization and SMBO.
Grid search 
implements all possible combination of hyperparameters for a learner and tries to find out the best one. It suffers if data have high dimensional space called the “curse of dimensionality”. It tries all combinations but only a few of the tuning parameters really matter.
Random search 
sets up a grid of hyperparameter values and select random combinations to train the model and evaluate. The evaluation is based on a specified probability distribution. The main problem of this method is at each step, it does not use information from the prior steps.
In contrast to Grid or Random search, Bayesian optimization  keeps track of past evaluation results and use them to build a probabilistic model mapping hyperparameters to a probability of a score on the objective function . This probabilistic model is called “surrogate” for the objective function. The idea is to find the next set of hyperparameters to evaluate on the actual objective function by selecting hyperparameters that perform best on the surrogate function.
Sequential model-based optimization (SMBO)  is a formalization of Bayesian optimization. It runs trials one by one sequentially, each time trying better hyperparameters using Bayesian reasoning and updating the surrogate model .
Recent studies have shown that hyperparameter optimization can achieve better performance than using “off-the-shelf” configurations in several research areas in software engineering, e.g., software effort estimation and software defect prediction. We are first to apply hyperparameter optimization in software fairness domain.
Iv-B FLASH: A Fast Sequential Model-Based Method
Nair et al.  proposed a fast SMBO approach called FLASH for multiobjective optimization. FLASH’s acquisition function uses Maximum Mean. Maximum Mean returns the sample (configuration) with the highest expected (performance) measure. FLASH models each objective as a separate performance (CART) model. Because the CART model can be trained for one performance measure or dependent value. Nair reports that FLASH runs orders of magnitude faster than NSGA-II, but that was for software configuration problems. This work is the first study to try using FLASH to optimize for learner performance while at the same time improving fairness.
|Good Credit||Bad Credit|
V-a RQ1: Does optimizing for fairness damage model prediction performance ?
We have verified our method along with four other related works to answer this question. Table I shows the datasets we used. We randomly divided them into three sets - training (70%), validation (15%) and test (15%). Prior researchers who worked with these datasets have used Logistic Regression as classification model [27, 28, 29]. We also decided to use this learner. Before moving to results, here we briefly describe prior works which we selected for our study. There are mainly three kinds of prior works -
Pre-processing algorithms: In this method, data is pre-processed(before classification) in such a way that discrimination is reduced. Kamiran et al. proposed Reweighing  method that generates weights for the training examples in each (group, label) combination differently to ensure fairness. Later, Calmon et al. proposed an Optimized pre-processing method  which learns a probabilistic transformation that edits the labels and features with individual distortion and group fairness.
In-processing algorithms: This is an optimization approach where dataset is divided into train, validation and test set. After learning from training data, model is optimized on the validation set and finally applied on the test set. Our Hyperparameter Optimization using FLASH approach lies into this category. Zhang et al. proposed Adversarial debiasing  method which learns a classifier to maximize accuracy and simultaneously reduce an adversary’s ability to determine the protected attribute from the predictions. This generates a fair classifier because the predictions cannot carry any group discrimination information that the adversary can exploit.
Post-processing algorithms: Hereafter classification, the class labels are changed to reduce discrimination. Kamiran et al. proposed Reject option classification approach  which gives unfavorable outcomes to privileged groups and favorable outcomes to unprivileged groups within a confidence band around the decision boundary with the highest uncertainty.
|Optimized Pre- processing||German||Age||0.70||0.68||0.66||0.58|
|FLASH optimizes for AOD & EOD||German||Age||0.74||0.68||0.20||0.45|
Table II shows the results of our approach (FLASH) and four algorithms from prior works. We see that there are a few gray cells and many black cells indicating that achieving fairness damages performance - which bolsters the conclusion made by Berk et al.. In summary, fairness can have a cost. Our next question checks if multiobjective optimization can better trade-off between performance and fairness.
V-B RQ2: Can we optimize machine learning model for both fairness and performance?
Here, we applied FLASH algorithm but this time, we considered four goals together: recall, false alarm, AOD, EOD. The first two are related to performance and second two are related to fairness. For recall, larger values are better while for everything else, smaller is better. For this part of our study, we used two learning models - logistic regression and CART.
We have chosen four hyperparameters for both the learners to optimize for. For logistic regression (C, penalty, solver, max_iter) and for CART - (criterion, splitter , min_samples_leaf, min_samples_split). Table III shows the results. The “Before” column shows results with no tuning and “After” column shows tuned results. We can see that for the German dataset, we improved three objectives and recall did not decrease. In the Adult dataset, we improved three objectives with minor damage of recall. With the Compas dataset, there was no improvement.
In summary, the results are clearly indicating if multiobjective optimization understand all the goals of learning (fairness and performance), then it is possible to achieve one without damaging the other. Our last research question asks what is the cost of this kind of optimization.
V-C RQ3. How much time does optimization take?
Default logistic regression takes 0.56s, 0.15s and 0.11s for Adult, Compas and German dataset respectively. When we apply hyperparameter optimization, the cumulative time for training, tuning and testing become 16.33s, 4.34s and 3.55s for those datasets. We assert that runtimes of less than 20 seconds is a relatively small price to pay to ensure fairness.
As to larger, more complex problems, Nair et al.  reports that FLASH scales to problems with larger order of magnitude than other optimizers. It is a matter for future research to see if such scale is possible/required to handle fairness of SE data.
Vi Conclusion & Future Work
Our experiments show that it might be possible to make software fair, without compromising other design goals (like predictive performance). Like Brun et al. , we propose that software bias detection and mitigation should be included in the software life-cycle. In agile practices, before any release, software should go through fairness testing and mitigation.
In this study, we only considered logistic regression and CART decision tree. In the future, we will explore more learning models. Another area to explore in the future is more data sets. Here, we used the same three datasets used by other publications in this area. All these datasets are small and so may not be representative of other real world scenarios. Ideally, software companies should consider making their data available which they think might be beneficial for bias related study. Data is not the only challenge, domain knowledge is important to understand the significance of protected attributes. For example, can/should we train our model without any protected attributes at all (gender, race, age)? In our experience, domain experts have strong opinions on that matter.
-  Y. Brun and A. Meliou, “Software fairness,” ser. ESEC/FSE 18. NY, USA: ACM, pp. 754–759.
-  F. B. Aydemir and F. Dalpiaz, “A roadmap for ethics-aware software engineering,” ser. FairWare ’18. NY, USA: ACM, pp. 15–21.
-  “Google’s sentiment analyzer thinks being gay is bad,” Motherboard, Oct 2017. [Online]. Available: https://bit.ly/2yMax8V
“Study finds gender and skin-type bias in commercial artificial-intelligence systems.” [Online]. Available:https://bit.ly/2LxosK6
-  “Google apologizes for mis-tagging photos of african americans,” July 2015. [Online]. Available: https://cbsn.ws/2LBYbdy
-  “Machine bias,” www.propublica.org, May 2016. [Online]. Available: https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
-  “Amazon scraps secret ai recruiting tool that showed bias against women,” Oct 2018. [Online]. Available: https://reut.rs/2Po4ZJi
-  C. O’Neil, Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. Crown Publishing Group, 2016.
-  R. Angell, B. Johnson, Y. Brun, and A. Meliou, “Themis: Automatically testing software for discrimination,” ser. ESEC/FSE 18.
-  S. Galhotra, Y. Brun, and A. Meliou, “Fairness testing: testing software for discrimination,” ESEC/FSE 17.
-  S. Udeshi, P. Arora, and S. Chattopadhyay, “Automated directed fairness testing,” ASE18.
-  A. Albarghouthi and S. Vinitsky, ser. FAT* ’19. NY, USA: ACM, pp. 211–219.
-  F. Tramer, V. Atlidakis, R. Geambasu, D. Hsu, J.-P. Hubaux, M. Humbert, A. Juels, and H. Lin, “Fairtest: Discovering unwarranted associations in data-driven applications,” EuroS&P17, Apr.
-  L. Breiman, Classification and regression trees. Routledge, 2017.
-  T. Xia, R. Krishna, J. Chen, G. Mathew, X. Shen, and T. Menzies, “Hyperparameter optimization for effort estimation,” arXiv preprint arXiv:1805.00336, 2018.
-  H. Osman, M. Ghafari, and O. Nierstrasz, “Hyperparameter optimization to improve bug prediction accuracy,” in MaLTeSQuE. IEEE, 2017.
-  J. Martin, “Bias or systematic error (validity),” 2010. [Online]. Available: https://www.ctspedia.org/do/view/CTSpedia/BiasDefinition
-  R. Bellamy, K. Dey, M. Hind, S. C. Hoffman, S. Houde, K. Kannan, P. Lohia, J. Martino, S. Mehta, A. Mojsilovic, S. Nagar, K. Natesan Ramamurthy, J. Richards, D. Saha, P. Sattigeri, M. Singh, R. Kush, and Y. Zhang, “Ai fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias,” 10 2018.
-  A. Biedenkapp, K. Eggensperger, T. Elsken, S. Falkner, M. Feurer, M. Gargiani, F. Hutter, A. Klein, M. Lindauer, I. Loshchilov et al., “Hyperparameter optimization,” Artificial Intelligence, 2018.
-  L. Franceschi, M. Donini, P. Frasconi, and M. Pontil, “Forward and reverse gradient-based hyperparameter optimization,” arXiv preprint arXiv:1703.01785, 2017.
-  J. S. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl, “Algorithms for hyper-parameter optimization,” in NIPS11, pp. 2546–2554.
-  J. Bergstra and Y. Bengio, “Random search for hyper-parameter optimization,” Journal of Machine Learning Research, 2012.
-  M. Pelikan, D. E. Goldberg, and E. Cantú-Paz, “Boa: The bayesian optimization algorithm,” in GECCO99, 1999, pp. 525–532.
-  W. Koehrsen, “A conceptual explanation of bayesian hyperparameter optimization for machine learning,” 2018.
-  F. Hutter, H. H. Hoos, and K. Leyton-Brown, “Sequential model-based optimization for general algorithm configuration,” in Learning and Intelligent Optimization. Springer Berlin Heidelberg, 2011.
-  V. Nair, Z. Yu, T. Menzies, N. Siegmund, and S. Apel, “Finding faster configurations using flash,” TSE, pp. 1–1, 2018.
-  T. Kamishima, S. Akaho, H. Asoh, and J. Sakuma, “Fairness-aware classifier with prejudice remover regularizer,” in Machine Learning and Knowledge Discovery in Databases. Springer Berlin Heidelberg, 2012.
-  F. Calmon, D. Wei, B. Vinzamuri, K. Natesan Ramamurthy, and K. R. Varshney, “Optimized pre-processing for discrimination prevention,” in NIPS17.
M. Hardt, E. Price, and N. Srebro, “Equality of opportunity in supervised learning,” 10 2016.
-  F. Kamiran and T. Calders, “Data preprocessing techniques for classification without discrimination,” KAIS12.
-  B. H. Zhang, B. Lemoine, and M. Mitchell, “Mitigating unwanted biases with adversarial learning,” ser. AIES ’18. NY, USA: ACM.
-  F. Kamiran, S. Mansha, A. Karim, and X. Zhang, “Exploiting reject option in classification for social discrimination control,” Inf. Sci., 2018.
-  R. Berk, H. Heidari, S. Jabbari, M. Joseph, M. Kearns, J. Morgenstern, S. Neel, and A. Roth, “A convex framework for fair regression,” 2017.