Fairness in Learning-Based Sequential Decision Algorithms: A Survey

by   Xueru Zhang, et al.
University of Michigan

Algorithmic fairness in decision-making has been studied extensively in static settings where one-shot decisions are made on tasks such as classification. However, in practice most decision-making processes are of a sequential nature, where decisions made in the past may have an impact on future data. This is particularly the case when decisions affect the individuals or users generating the data used for future decisions. In this survey, we review existing literature on the fairness of data-driven sequential decision-making. We will focus on two types of sequential decisions: (1) past decisions have no impact on the underlying user population and thus no impact on future data; (2) past decisions have an impact on the underlying user population and therefore the future data, which can then impact future decisions. In each case the impact of various fairness interventions on the underlying population is examined.



page 1

page 2

page 3

page 4


On the Long-term Impact of Algorithmic Decision Policies: Effort Unfairness and Feature Segregation through Social Learning

Most existing notions of algorithmic fairness are one-shot: they ensure ...

A survey of algorithmic recourse: definitions, formulations, solutions, and prospects

Machine learning is increasingly used to inform decision-making in sensi...

Achieving Fairness in Determining Medicaid Eligibility through Fairgroup Construction

Effective complements to human judgment, artificial intelligence techniq...

Identifying Optimal Sequential Decisions

We consider conditions that allow us to find an optimal strategy for seq...

Threat Modeling Data Analysis in Socio-technical Systems

Our decision-making processes are becoming more data driven, based on da...

Enhancing the Accuracy and Fairness of Human Decision Making

Societies often rely on human experts to take a wide variety of decision...

Algorithmic Fairness Datasets: the Story so Far

Data-driven algorithms are being studied and deployed in diverse domains...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Decision-making algorithms that are built from real-world datasets have been widely used in various applications. When these algorithms are used to inform decisions involving human beings (e.g., college admission, criminal justice, resume screening), which are typically done by predicting certain variable of interest from observable features, they may inherit the potential, pre-existing bias in the dataset and exhibit similar discrimination against protected attributes such as race and gender. For example, the COMPAS algorithm used by courts for predicting recidivism in the United States has been shown to be biased against black defendants Dressel and Farid (2018); job searching platform XING ranks less qualified male applicants higher than female applicants who are more qualified Lahoti et al. (2019); a nationwide algorithm used for allocating medical resources in US is biased against black patients Obermeyer and Mullainathan (2019).

There are various potential causes for such bias. It may have been introduced when data is collected. For instance, if data sampled from a minority group is much smaller in size than that from a majority group, then the model could be more in favor of the majority group due to this representation disparity (e.g., more than a third of data in ImageNet and Open Images, two datasets widely used in machine learning research communities, is US-based

Shankar et al. (2017)). Another example is when the data collection decision itself reflects bias, which then impacts the collected data (e.g., if more police officers are dispatched to places with higher crime rate to begin with, then crimes are more likely to be recorded in these places Ensign et al. (2018)). Even when the data collection process is unbiased, bias may already exist in the data. Historical prejudice and stereotypes can be preserved in data (e.g., the relationship between ”man” and ”computer programmers” were found to be similar to that between ”woman” and ”homemaker” Bolukbasi et al. (2016)). An interested reader can find more detailed categorization of bias in the survey Mehrabi et al. (2019).

The problem does not merely stop here. On one hand, decisions made about humans can affect their behavior and reshape the statistics of the underlying population. On the other hand, decision-making algorithms are updated periodically to assure high performance on the targeted populations. This complex interplay between algorithmic decisions and the underlying population can lead to pernicious long term effects by allowing biases to perpetuate and reinforcing pre-existing social injustice. For example, Aneja and Avenancio-León (2019) shows that incarceration can significantly reduce people’s access to finance, which in turn leads to substantial increase in recidivism; this forms a credit-driven crime cycle. Another example is speech recognition: products such as Amazon’s Alexa and Google Home are shown to have accent bias with native speakers experiencing much higher quality than non-native speakers Harwell (2018)

. If this difference in user experience leads to more native speakers using such products while driving away non-native speakers, then over time the data used to train the algorithms may become even more skewed toward native speakers, with fewer and fewer non-native samples. Without intervention, the resulting model may become even more accurate for the former and less for the latter, which then reinforces their respective user experience

Hashimoto et al. (2018). Similar negative feedback loops have been observed in various settings such as recommendation system Chaney et al. (2018), credit market Fuster et al. (2018), and policing prediction Ensign et al. (2018). Preventing discrimination and guaranteeing fairness in decision-making is thus both an ethical and a legal imperatives.

To address the fairness issues highlighted above, a first step is to define fairness. Anti-discrimination laws (e.g., Title VII of the Civil Rights Act of 1964) typically assess fairness based on disparate impact and disparate treatment. The former happens when outcomes disproportionately benefit one group while the latter occurs when the decisions rely on sensitive attributes such as gender and race. Similarly, various notions of fairness have been formulated mathematically for decision-making systems and they can be categorized roughly into two classes:

  • Individual fairness: this requires that similar individuals are treated similarly.

  • Group fairness: this requires (approximate) parity of certain statistical measures (e.g., positive classification rate, true positive rate, etc.) across different demographic groups.

In Section 2 we present the definitions of a number of commonly used fairness measures. Their suitability for use is often application dependent, and many of them are incompatible with each other Kleinberg et al. (2017).

To satisfy the requirement of a given definition of fairness, various approaches have been proposed and they generally fall under three categories:

  1. Pre-processing: by changing the original dataset such as removing certain features, reweighing and so on, e.g., Calders and Žliobaitė (2013); Kamiran and Calders (2012); Zemel et al. (2013); Gordaliza et al. (2019).

  2. In-processing: by modifying the decision-making algorithms such as imposing fairness constraints or changing objective functions, e.g., Berk et al. (2017); Zafar et al. (2017, 2019); Agarwal et al. (2018).

  3. Post-processing: by adjusting the output of the algorithms based on sensitive attributes, e.g., Hardt et al. (2016).

While the effectiveness of these approaches have been shown in various domains, most of these studies are done using a static framework where only the immediate impact of the learning algorithm is assessed but not its long-term consequences. Consider an example where a lender decides whether or not to issue a loan based on the applicant’s credit score. Decisions satisfying an identical true positive rate (equal opportunity) across different racial groups can make the outcome seem fairer Hardt et al. (2016). However, this can potentially result in more loans issued to less qualified applicants in the group whose score distribution skews toward higher default risk. The lower repayment among these individuals causes their future credit scores to drop, which moves the score distribution of that group further toward higher default risk Liu et al. (2018). This shows that intervention by imposing seemingly fair decisions in the short term can lead to undesirable results in the long run Zhang et al. (2019a). As such, it is critical to understand the long-term impacts of fairness interventions on the underlying population when developing and using such decision systems.

In this survey, we focus on fairness in sequential decision systems. We introduce the framework of sequential decision-making and commonly used fairness notions in Section 2. The literature review is done in two parts. We first consider sequential settings where decisions do not explicitly impact the underlying population in Section 3, and then consider sequential settings where decisions and the underlying population interact with each other in Section 4. The impact of fairness interventions is examined in each case. For consistency of the survey, we may use a set of notations different from the original works.

2 Preliminaries

2.1 Sequential Decision Algorithms

The type of decision algorithms surveyed in this paper are essentially classification/prediction algorithms used by a decision maker to predict some variable of interest (label) based on a set of observable features. For example, judges predict whether or not a defendant will re-offend based on its criminal records; college admission committee decides whether or not to admit an applicant based on its SAT; lender decides whether or not to issue a loan based on an applicant’s credit score.

To develop such an algorithm, data are collected consisting of both features and labels, from which the best mapping (decision rule) is obtained, which is then used to predict unseen, new data points. Every time a prediction is made, it can either be correct (referred to as a gain) or incorrect (referred to as a loss). The optimal decision rule without fairness consideration is typically the one that minimizes losses or maximizes gains.

In a sequential framework, data arrive and are observed sequentially and there is feedback on past predictions (loss or gain), and we are generally interested in optimizing the performance of the algorithm over a certain time horizon. Such a sequential formulation roughly falls into one of two categories.


The goal of the algorithm is to learn a near-optimal decision rule quickly, noting that at each time step only partial information is available, while minimizing (or maximizing) the total loss (or gain) over the entire horizon. Furthermore, within the context of fairness, an additional goal is to understand how a fairness requirement impacts such a decision rule.


This is a setting where not only do data arrival sequentially, but decisions made in the past can affect the feature space of the underlying population, thereby changing the nature of future observations. The goal in this case is to learn an optimal decision rule at each time step and understand the impact it has on the population and how fairness requirement further adds to the impact.

2.2 Notions of Fairness

As mentioned in Section 1

, different notions of fairness can be generally classified into

individual fairness and group fairness.

Group fairness: For simplicity of exposition and without loss of generality, we will limit ourselves to the case of two demographic groups , , distinguished based on some sensitive attribute

representing group membership (e.g., gender, race). Group fairness typically requires certain statistical measure to be equal across these groups. Mathematically, denote by random variable

an individual’s true label and its prediction generated from a certain decision rule. Then the following is a list of commonly used group fairness criteria.

  1. Demographic Parity (DP): it requires the positive prediction rate be equal across different demographic groups, i.e., .

  2. Equal of Opportunity (EqOpt): it requires true positive rate (TPR)111Based on the context, this criterion can also refer to equal false negative rate (FNR), false positive rate (FPR), or true negative rate (TNR) be equal across different demographic groups, i.e., .

  3. Equalized Odds (

    EO): it requires both the false positive rate and true positive rate be equal across different demographic groups, i.e., .

  4. Equalized Loss (EqLos): it requires different demographic groups experience the same total prediction error, i.e., .

Individual fairness: Such a criterion targets the individual, rather than group level. Commonly used examples are as follows.

  1. Fairness through awareness (FA): this requires that similar individuals be treated similarly.

  2. Meritocratic fairness (MF): this requires that less qualified individuals not be favored over more qualified individuals.

The above definitions do not specify how similarity among individuals or qualification of individuals are measured, which can be context dependent.

3 (Fair) Sequential Decision When Decisions Do Not Affect Underlying Population

Fairness definition Data type Problem type
Group fairness Individual fairness
Heidari and Krause (2018) FA i.i.d. P1
Gupta and Kamble (2019) FA non-i.i.d. P1
Bechavod et al. (2019) EqOpt i.i.d. P1
Joseph et al. (2016) MF i.i.d. P1
Joseph et al. (2018) MF i.i.d. P1
Liu et al. (2017) FA i.i.d. P1
Valera et al. (2018) i.i.d. P1
Blum et al. (2018) EqOpt, EO, EqLos non-i.i.d. P1
Chen et al. (2019) i.i.d. P1
Li et al. (2019) i.i.d. P1
Patil et al. (2019) i.i.d. P1
Gillen et al. (2018) FA non-i.i.d. P1
Table 1: Summary of related work when decisions do not affect the underlying population. represents the use of fairness definitions or interventions not included in Section 2.2.

We first focus on a class of sequential decision problems (P1) where the decision at each time step does not explicitly affect the underlying population; a list of these studied are summarized in Table 1. Most of these works have developed algorithms that can learn a decision rule with sufficient accuracy/performance subject to certain fairness constraint, and the impact of fairness on these sequential decision-making problems is reflected through its (negative) effect on the achievable performance.

3.1 Bandits, Regret, and Fair Regret

We begin with Heidari and Krause (2018); Gupta and Kamble (2019); Bechavod et al. (2019) on online learning problems, where a decision maker at each time receives data from one individual and makes decision according to some decision rule. It then observes the loss (resp. utility) incurred from that decision. The goal is to learn a decision rule from a set of data collected over time steps under which (1) the accumulated expected loss (resp. utility) over steps is upper (resp. lower) bounded; and (2) certain fairness criterion is satisfied. Specifically, Heidari and Krause (2018); Gupta and Kamble (2019)

focus on individual fairness which ensures that similar individuals (who arrive at different time steps) be treated similarly, by comparing each individual with either all individuals within a time epoch

Heidari and Krause (2018); Gupta and Kamble (2019) or only those who’ve arrived in the past Gupta and Kamble (2019). By contrast, Bechavod et al. (2019) focuses on group fairness (EqOpt), where at each time the arriving individual belongs to one demographic group and the goal is to ensure different demographic groups in general receive similar performance over the entire time horizon. Moreover, Bechavod et al. (2019) considered a partial feedback scenario where the loss (resp. utility) is revealed to the decision maker only when certain decisions are made (e.g., whether an applicant is qualified for a certain job is only known when he/she is hired). In each of these settings, the impact of fairness constraint on accumulated expected loss/utility is examined and quantified and an algorithm that satisfies both (approximate) fairness and certain loss/utility is developed.

In some applications, the decision maker at each time makes a selection from multiple choices. For example, hiring employees from multiple demographic groups, selecting candidates from a school for certain competitions, etc. Specifically, the decision maker at each time receives features of multiple individuals (potentially from different demographic groups) and the corresponding sequential decision problems can be formulated as a multi-armed bandit problem, where each arm represents either one specific individual or one demographic group and choosing an arm represents selecting one individual (from one demographic group). In a classic stochastic bandit problem, there is a set of arms . The decision maker selects an arm at time from and receives a random reward , drawn from a distribution with unknown mean .

Let represent all history information received by the decision maker up to time . Then the decision rule at

is a probability distribution over all arms. Denote by

the probability of selecting arm at time given history . The regret of applying the decision rule over time steps is defined as:

The goal of a fair decision maker in this context is to select such that the regret over time steps is minimized, while certain fairness constraint is satisfied.

Joseph et al. in Joseph et al. (2016) proposed the use of meritocratic fairness in the above bandit setting as follows. Consider a multi-armed bandit problem where each arm represents an individual and the decision maker selects one individual at each time. Let the mean reward represent the average qualification of individuals (e.g., hiring more qualified applicant can bring higher benefit to a company); then it is unfair if the decision maker preferentially chooses an individual less qualified in expectation over another. Formally, the decision maker is defined to be -fair over time steps if with probability , for all pairs of arms and , the following holds.


Joseph et al. (2016) developed an algorithm to find optimal decision rules in classic stochastic setting that is -fair. To ensure -fairness, for any two arms , they should be selected with equal probability unless . Let , be the upper and lower confidence bounds of arm at time . Then arms and are linked if ; arms and are chained if they are in the same component of the transitive closure of the linked relation. The algorithm in Joseph et al. (2016) first identifies the arm with the highest upper confidence bound and finds all arms chained to it (). For arms not in , the decision maker has sufficient confidence to claim they are less qualified than others, while for arms in , the decision maker randomly selects one at uniform to ensure fairness.

Joseph et al. (2016) shows that if , then the algorithm can achieve . In contrast, without fairness consideration, the original upper confidence bound (UCB) algorithm proposed by Auer et al. Auer et al. (2002) achieves regret , where is the difference between the expected rewards of the optimal arm and a sub-optimal arm. The cubic dependence on (the number of arms) in the former is due to the fact that any fair decision rule must experience constant per-step regret for steps on some instances, i.e., the average per-step regret is for .

The idea of this chaining strategy can also be adapted to develop fair algorithms for more general scenarios such as contextual bandit problems Joseph et al. (2016) and bandits with different (or even infinite) number of arms at each time among which multiple arms can be selected Joseph et al. (2018). Similar to constraint (1), fairness metrics in these generalized settings are also defined in terms of individual’s expected qualification, and stipulate that two similar individuals with the same expected reward be treated similarly, even though their reward distributions can be significantly different.

In contrast, Liu et al. Liu et al. (2017) proposes smooth fairness based on individuals’ reward distributions rather than expected reward, which requires that individuals with similar reward distributions be selected with similar probabilities. Formally, and , the decision rule is -smooth fair w.r.t. a divergence function , if and for any pair of arms , the following holds with probability at least :



denotes a Bernoulli distribution with parameter


Compared with meritocratic fairness, smooth fairness is weaker in the sense that it allows a worse arm to be selected with higher probability. To quantify such violation, Liu et al. (2017) further proposes a concept of fairness regret, where a violation occurs when the arm with the highest reward realization at a given time is not selected with the highest probability. Based on this idea, the fairness regret of decision rule at time is defined as

and the cumulative fairness regret is defined as , where is the probability that the reward realization of arm is the highest among all arms.

Two algorithms were developed in Liu et al. (2017) for special types of bandit problems: (1) Bernoulli bandit, where the reward distributions satisfy ; and (2) Dueling bandit: the decision maker selects two arms and only observes the outcome . These algorithms satisfy smooth fairness w.r.t. total variation distance with low fairness regret.

In satisfying FA that similar individuals be treated similarly, one challenge is to define the appropriate context-dependent metric to quantify ”similarity”. Most studies in this space assume such a metric is given. Gillen et al. (2018) proposes to learn such a similarity metric from the decision process itself. Specifically, it considers a linear contextual bandit problem where each arm corresponds to an unknown parameter . At each time the decision maker observes arbitrarily and possibly adversarially selected contexts from arms, each representing features of an individual. It selects one (say arm ) among them according to some decision rule and receives reward with mean . Gillen et al. (2018) focuses on individual fairness that individuals with similar contexts (features) be selected with similar probabilities, i.e., , , for some unknown metric . Similar to Liu et al. (2017), Gillen et al. (2018) also defines a fairness regret to quantify fairness violation over time steps. Specifically, let be the total number of arm pairs violating -fairness and the total fairness regret over steps is , where represents the error tolerance. The goal is to find a decision rule with low fairness regret that is also near-optimal (w.r.t. the best fair decision rule).

However, since is unknown, to achieve the above objective, also needs to be learned. To do so, it assumes that in addition to reward , the decision maker at each time receives feedback , i.e., the set of all pairs of individuals for which the decision rule violates the fairness constraint. With such (weak) feedback, a computationally efficient algorithm is developed in Gillen et al. (2018) that for any metric following the form of Mahalanobis distance, i.e., for some matrix , any time horizon and any , with high probability it (i) obtains regret w.r.t. the best fair decision rule; and (ii) violates unknown fairness constraints by more than on at most steps.

Other studies, such as Chen et al. (2019); Li et al. (2019); Patil et al. (2019) also use a bandit formulation with fairness consideration, where the fairness constraint requires either each arm be pulled for at least a certain fraction of the total available steps, or the selection rate of each arm be above a threshold. Algorithms that satisfy both (approximate) fairness and low regret are developed in these studies.

3.2 Fair Experts and Expert Opinions

In some sequential decision problems, decision maker at each time may follow advice from multiple experts and at each it selects expert according to a decision rule where denote the probability of selecting expert at time . Blum et al. Blum et al. (2018) considers a sequential setting where at each time a set of experts all make predictions about an individual (possibly based on sensitive attribute ). Let be the individual’s true label and expert ’s prediction be , then the corresponding loss of expert is measured as . By following decision rule , the decision maker takes ’s advice with probability , and the overall expected loss at time is given by . The decision maker is assumed to observe , and .

In Blum et al. (2018), each expert in isolation is assumed to satisfy certain fairness criterion over a horizon. Specifically, given a sequence of individuals , let be the set of time steps at which corresponding individuals are from and have label , expert satisfies EqOpt if holds. The decision maker following is said to be -fair w.r.t. EqOpt if the following holds,

Similar formula can be derived for the EO and EqLos criteria. The goal of the decision maker is to find -fair w.r.t. from a set of fair experts that all satisfy fairness in isolation, and at the same time perform as (approximate) good as the best expert in hindsight. Formally, define -approximate regret of over time steps with respect to decision maker as follows:


Then the goal is to achieve vanishing regret , and .

When the input is i.i.d., the above setting is trivial because the best expert can be learned in rounds and the decision maker can follow its advice afterwards. Because each expert is fair in isolation, this also guarantees vanishing discrimination.

However, when input is non-i.i.d., achieving such objective is challenging. Blum et al. (2018) considers an adversarial setting where both and can be adaptively chosen over time according to . It first examines the property of EqOpt and shows that given a set of experts that satisfies EqOpt, it is impossible to find a decision rule with vanishing regret that can also preserve -fairness w.r.t. EqOpt. This negative result holds for both the cases when group identity information is used in determining (group-aware) and the cases when the group information is not used (group-unaware). Specifically, for both cases, Blum et al. (2018) constructs scenarios (about how an adversarial selects over time) under which for any that is smaller than a constant , such that for any that satisfies , , violates the -fairness w.r.t. EqOpt.

Since EqOpt is strictly weaker than EO, the above impossibility result in EqOpt naturally generalizes to EO. In contrast, under EqLos, given a set of experts that satisfies EqLos fairness, , there exists group-aware that can simultaneously attain -fairness and the vanishing regret. The idea is to run two separate multiplicative weights algorithms for two groups. Because one property of the multiplicative weights algorithm is that it performs no worse than the best expert in hindsight but also no better. Therefore the average performance of each group is approximately equal to the average performance attained by the best expert for that group. Because each expert is EqLos fair, the average performance attained by best experts of two groups are the same. Consequently, both vanishing regret and -fairness are satisfied. This positive result is due to the consistency between performance and fairness measure for EqLos. However, such positive result does not generally hold for EqLos. If only one multiplicative algorithm is performed without separating two groups, i.e., run in group-unaware manner, then it can be shown that and , any algorithm satisfying vanishing regret also violates -fairness w.r.t. EqLos.

Valera et al. Valera et al. (2018) studied a matching problem in sequential framework, where a set of experts need to make predictions about individuals from two demographic groups over time steps, where at time step individual ’s decision is made by expert . Different from Blum et al. (2018) where experts are all fair (w.r.t. a particular metric) over a horizon and at each time only one expert’s advice is followed on one individual, experts in Valera et al. (2018) can be biased and at each time predictions from decision makers are all used and each is assigned to one individual. The algorithms for finding the optimal assignments are developed for cases with and without fairness intervention, which can improve both the overall accuracy and fairness as compared to random assignment, and fairness is guaranteed even when a significant percentage (e.g., 50%) of experts are biased against certain groups.

3.3 Fair Policing

Ensign et al. Ensign et al. (2018) studied a predictive policing problem, where the decision maker at each time decides how to allocate patrol officers to different areas to detect crime based on historical crime incident data. The goal is to send officers to each area in numbers proportional to the true underlying crime rate of that area, i.e., areas with higher crime rate are allocated more officers. Ensign et al. (2018) first characterizes the long-term property of existing predictive policing strategies (e.g., PredPol software), in which more officers are sent to areas with the higher predicted crime rates and the resulting incident data is fed back into the system. By modeling this problem using urn model, Ensign et al. (2018) shows that under such a method, one area can eventually consume all officers, even though the true crime rates may be similar across areas. This is because by allocating more officers to an area, more crimes are likely to be detected in that area; allocating more officers based on more detected crimes is thus not a proper method. To address this issue, effective approaches are proposed in Ensign et al. (2018), e.g., by intentionally normalizing the detected crime rates according to the rates at which police are sent.

4 (Fair) Sequential Decision When Decisions Affect Underlying Population

Fairness notion Problem type
Group fairness Individual fairness
Liu et al. (2018) EqOpt, DP P2
Heidari et al. (2019) P2
Kannan et al. (2019) EqOpt, P2
Mouzannar et al. (2019) DP P2
Liu et al. (2019) P2
Hu and Chen (2018) DP, P2
Hashimoto et al. (2018) P2
Zhang et al. (2019b) EqOpt, DP P2
Jabbari et al. (2017) MF P1

Wen et al. (2019)
Table 2: Summary of related work when decisions affect underlying population. represents some other fairness notions or interventions that are not introduced in Section 2.2.

We next examine a second class of sequential decision problems (P2) where the decisions affect the underlying population; a list of these studied are summarized in Table 2. We will start with a set of papers that use a two-stage model, followed by a set of papers focusing on finite-horizon and infinite-horizon models.

4.1 Two-Stage Models

To examine the long term impact of fairness intervention on the underlying population, some studies Liu et al. (2018); Heidari et al. (2019); Kannan et al. (2019) construct two-stage models, whereby the first stage decisions (under certain fairness criterion) are imposed on individuals from two demographic groups , , which may cause individuals to take certain actions, and the overall impact of this one-step intervention on the entire group is then examined in the second stage.

Let be the size of as the fraction of the entire population and . Liu et al. (2018) focuses on a one-dimensional setting where an individual from either group has feature with and sensitive attribute representing his/her group membership. Let be ’s feature distribution and the individual’s true label. The decision maker makes predictions on individuals using the decision rule and receives expected utility for making a positive prediction of an individual with feature (e.g., average profit of a lender by issuing a loan to applicants whose credit score is 760). The expected utility of the decision maker under is given by:

Define the selection rate of under a decision rule as . Then given feature distributions, the relationship between and can be described by an invertible mapping so that and .

In Liu et al. (2018), decision rules for , are selected such that is maximized under fairness constraints defined as follows:

  • Simple: it requires the same decision rule be used by , , i.e., .

  • Demographic Parity (DP): it requires the selection rates of , are equalized, i.e., .

  • Equal of Opportunity (EqOpt): it requires the true positive rate (TPR) of , are equalized, i.e., .

Once an individual with feature is predicted as positive () in the first stage, its feature may be affected; denote the average of such change as . For example, consider a lending scenario where a lender decides whether or not to issue loans to applicants based on their credit scores. Among applicants who are issued loans, those with the higher (resp. lower) credit score are more likely to repay (resp. default); as a result, the credit scores may increase for applicants who can repay the loans () but decrease for those who default (). Consequently, the feature distribution of the entire group can be skewed. Let the impact of a decision rule on be captured by the average change of in , defined as . It can be shown that is a concave function in the selection rate .

Let the optimal fair decision rule that maximizes under fairness criterion be noted as , and the corresponding selection rate be noted as . Let group labels be assigned such that is the disadvantaged group in the sense that . Given , a decision rule causes

  • active harm to if ;

  • relative harm if ;

  • relative improvement if .

Due to the one-to-one mapping between the decision rule and the selection rate, the notation in the following is simplified as . Let be the harmful threshold for such that ; let be the max-improvement threshold such that ; let be the complementary threshold such that and .

The goal of Liu et al. (2018) is to understand the impact of imposing DP or EqOpt fairness constraint on , whether these fairness interventions can really benefit the disadvantaged group as compared to the Simple decision rule.

Liu et al. (2018) first examined the impact of Simple decision rule, and showed that if , then Simple threshold does not cause active harm, i.e., . In lending example, the condition means that the lender takes a greater risk by issuing a loan to an applicant than the applicant does by applying.

For DP and EqOpt fairness, Liu et al. (2018) showed that both could cause relative improvement, relative harm and active harm, under different conditions. We summarize these results below, for ,

  1. Under certain conditions, there exists such that , causes relatively improvement, i.e., .

  2. Under certain conditions, positive predictions can be over-assigned to for satisfying . There exists such that , causes relatively harm or active harm, i.e., or .

These results show that although it seems fair to impose DP and EqOpt constraints on decisions (e.g., by issuing more loans to the disadvantaged group), it may have unintended consequences and harm the disadvantaged group (e.g., features of disadvantaged group may deteriorate after being selected).

Liu et al. (2018) makes further comparisons between DP and EqOpt fairness. Generally speaking, DP and EqOpt cannot be compared in terms of . Because there exist both settings when DP causes harm while EqOpt causes improvement, and settings when EqOpt causes harm while DP causes improvement. However, for some special cases when and satisfy a specific condition, there exists such that , DP causes active harm while EqOpt causes improvement. Moreover, if under Simple decision rule, and hold, then can be satisfied, i.e., EqOpt can cause relative harm by selecting less than Simple rule.

An interested reader is referred to Liu et al. (2018) for details of the specific conditions mentioned above. It shows that temporal modeling and a good understanding of how individuals react to decisions are necessary to accurately evaluate the impact of different fairness criteria on the population.

Effort-based Fairness

Essentially, the issues of unfairness described in the preceding section may come from the fact that different demographic groups have different feature distributions, leading to different treatments. However, this difference in feature distributions is not necessarily because one group is inherently inferior to another; rather, it may be the result of the fact that advantaged group can achieve better features/outcomes with less effort. For example, if changing one’s school type from public to private can improve one’s SAT score, then such change would require much higher effort for the low-income population. From this point of view, Heidari et al. Heidari et al. (2019) proposes an effort-based notion of fairness, which measures unfairness as the disparity in the average effort individuals from each group have to exert to obtain a desirable outcome.

Consider a decision maker who makes a prediction about an individual using decision rule based on its

-dimensional feature vector

. Let be the individual’s true label, its sensitive attribute, and the predicted label. Define a benefit function that quantifies the benefit received by an individual with feature and label if he/she is predicted as .

For an individual from who changes his/her data from to , the total effort it needs to take is measured as , where , and denotes the effort needed for an individual from to change its th feature from to . Accordingly, the change in the individual’s benefit by making such an effort is , and the total utility received by an individual from in changing his/her data is

Define as the expected highest utility can possibly reach by exerting effort. Heidari et al. (2019) suggests the use of the disparity between and as a measure of group unfairness.

The microscopic impact of decisions on each individual can be modeled using the above unfairness measure. Intuitively, if individuals can observe the behaviors of others similar to them, then they would have more incentive to imitate behaviors of those (social models) who receive higher benefit, as long as in doing so individuals receive positive utility.

Let be the training dataset representing samples of population in . Then can be regarded as a social model’s profile that an individual from aims to achieve, as long as . Given the change of each individual in , a new dataset in the next time step can be constructed accordingly.

Given , , the datasets before and after imposing decisions according to , the macroscopic impact of decisions on the overall underlying population can be quantified. Heidari et al. (2019) adopts the concept of segregation from sociology to measure the degree to which multiple groups are separate from each other. Specifically, the segregation of and are compared from three perspectives: Evenness, Clustering and Centralization. The details of each can be found in Heidari et al. (2019); here we only introduce Centralization as an example: this is measured as the proportion of individuals from a minority group whose prediction is above the average. The impact of decisions on the entire group is examined empirically by comparing Evenness, Clustering and Centralization of and .

Heidari et al. (2019) first trained various models

such as neural network, linear regressor and decision tree over a real-world dataset without imposing a fairness constraint. It shows that individuals by imitating social model’s data profile can either increase or decrease the segregation of the overall population, and different models may shift the segregation toward different directions. Next,

Heidari et al. (2019)

examined the impact of imposing fairness constraint on a linear regression model. Specifically, the fairness constraint requires each group’s average utility be above the same threshold, a higher threshold indicating a stronger fairness requirement. Empirical results show that segregation under different levels of fairness can change in completely different directions (decrease or increase), and impacts on

Evenness, Centralization and Clustering are also different.

Indeed, fairness intervention affects segregation in two competing ways. If more desirable outcomes are assigned to a disadvantaged group intentionally, then on one hand individuals from the disadvantaged group may have less motivation to change their features, on the other hand, the same individuals may serve as social models, which in turn can incentivize others from the same disadvantaged group to change their features. Both impacts are at play simultaneously and which one is dominant depends on the specific context. This paper highlights the fact that modifying decision algorithm is not the only way to address segregation and unfairness issues; imposing mechanisms before individuals enter the decision system may be another effective way, e.g., by decreasing the costs for individuals from the disadvantaged group to change their features.

A Two-Stage Model in College Admissions

Kannan et al. Kannan et al. (2019) studied a two-stage model in the case of college admissions and hiring. In the first stage, students from two demographic groups are admitted to a college based on their entrance exam scores; in the second stage an employer chooses to hire students from those who were admitted to the college based on their college grades. Specifically, let denote a student’s group membership and

his/her qualification drawn from a group-specific Gaussian distribution. Let variable

be the student’s entrance exam score with independent noise .

Denote by the college’s admissions decision about a student. Let be the admissions rule representing the probability a student from with score gets admitted, which is monotone non-decreasing in for . Consider a threshold decision rule of the following form:


For a student who is admitted, he/she receives a grade with ,

, where the variance

is determined by some grading rule. Specifically, can be regarded as a case where students’ grades are not revealed to the employer, whereas represents a case where the employer has perfect knowledge of the students’ qualifications. The employer decides whether or not to hire a student based on his/her grade. Let be the cost for the employer for hiring a student, which can either be known or unknown to the college. Then a student from with grade gets hired if the employer can achieve a non-negative expected utility, i.e., .

The goal of Kannan et al. (2019) is to understand what admission rules and grading rules should be adopted by the college in the first stage so that the following fairness goals may be attained in the second stage:

  • Equal of Opportunity (EqOpt): it requires the probability of a student being hired by the employer conditional on the qualification is independent of group membership .

  • Irrelevance of Group Membership (IGM): it requires the employer’s hiring decision, conditional on and , should be independent of group membership, i.e., , .

  • Strong Irrelevance of Group Membership (sIGM): it requires the employer’s posterior about students’ qualifications, conditional on and , should be independent of group membership, i.e., and , .

Below we present two simple scenarios found in Kannan et al. (2019), in which both EqOpt and IGM can be satisfied in the second phase under some admission rules.

  1. Noiseless entrance exam score, i.e., .

    In this scenario, the admission decision is determined by the student’s qualification completely. Kannan et al. (2019)

    shows that as long as the threshold in the admission decision rule is set as

    in Eqn. (4), then and with any grading rule, both EqOpt and IGM can be satisfied.

  2. No grade is revealed to the employer, i.e., .

    In this case, as long as the threshold in the admission decision rule is set as for some sufficiently large (e.g., highly selective MBA programs) in Eqn. (4), then , both EqOpt and IGM can be satisfied.

Kannan et al. (2019) also studied more general scenarios when noises and are both of finite variance, i.e., noisy entrance exam scores and when colleges report informative grades to the employer. When employer’s hiring cost is known to the college, , there always exist two thresholds , and a grade for college, under which always holds, i.e., IGM can always be satisfied.

However, if we consider the employer’s posterior distributions on students’ qualification, as long as two groups have different prior distributions, for any two thresholds in the admission rule, there always exists such that , i.e., satisfying sIGM is impossible.

Moreover, suppose prior distributions of two groups’ qualifications are Gaussian distributed with different mean but the same variance, then , there exists no threshold decision rule that can satisfy both EqOpt and IGM simultaneously. For EqOpt under some fixed hiring cost , in cases when grading rule has variance , there is no threshold decision rule such that EqOpt can be satisfied. For cases when , EqOpt can be satisfied only if the admission rule and grading rule can satisfy Such condition is generally impossible to hold. It concludes that EqOpt is generally is impossible to achieve.

If employer’s hiring cost is uncertain that college only knows the interval , when two groups have different priors, Kannan et al. (2019) shows that , neither EqOpt nor IGM can be satisfied even in isolation under a threshold admission rule.

The above results show that even with a simple model studied in Kannan et al. (2019), many common and natural fairness goals are impossible to achieve in general. Such negative results are likely to hold true in more complex models that capture more realistic aspects of the problem.

4.2 Long-Term Impacts on the Underlying Population

Decisions made about humans affect their actions. Bias in decisions can induce certain behavior, which is then captured in the dataset used to develop decision algorithms in the future. The work Liu et al. (2018); Heidari et al. (2019); Kannan et al. (2019) introduced in the previous section studied such one-step impact of decisions on the population. However, when newly developed algorithms are then used to make decisions about humans in the future, those humans will be affected and biases in the datasets generated by humans can perpetuate. This closed feedback loop becomes self-reinforcing and can lead to highly undesirable outcomes over time. In this section, we focus on the long-term impacts of decisions on population groups. The goal is to understand what happens to the underlying population when decisions and people interact with each other and what interventions are effective in sustaining equality in the long run.

Effects of Decisions on the Evolution of Features

One reason why decisions are made in favor of one group is that the favored group is believed to bring more benefit to the decision maker. For example, a lender issues more loans to a group believed to be more likely to repay, a company hires more from a group perceived to be more qualified, and so on. In other words, disparate treatment received by different groups is due to the disparity in their (perceived) abilities to produce good outcomes (qualifications). From this perspective, the ultimate social equality is attained when different demographic groups possess the same abilities/qualifications. In this section, we present studies reported in Mouzannar et al. (2019); Liu et al. (2019); Hu and Chen (2018) to understand how qualifications of different groups evolve over time under various fairness interventions, and under what conditions social equality may be attained.

Let , be two demographic groups, the size of as a fraction of the entire population and assumed constant, and . Each individual has feature , sensitive attribute , and label representing his/her qualification or the ability to produce certain good outcome. Define the qualification profile of at time as the probability distribution , . Changes in feature induced by decisions are captured by change in the qualification profile.

Using the definition of qualification profiles of two groups, social equality can be defined formally as equalized qualification profiles, i.e.,


Mouzannar et al. (2019); Liu et al. (2019) assume that the qualification profiles at each time are known to the decision maker, who makes prediction about each individual according to a decision rule and receives utility for making positive prediction , where and correspond to the loss and benefit, respectively. Define the selection rate of under a decision rule at time as . Then the expected utility of the decision maker at is:


Upon receiving a decision, a qualified individual can either remain qualified or become unqualified, and an unqualified individual can either become qualified or remain unqualified for the next time step. In Mouzannar et al. (2019), the evolution of a group’s qualification profile is modeled as a dynamical system as follows:


where represents the retention rate of subgroup who are qualified in time that are still qualified in , while represents the improvement rate of subgroup who are unqualified () at time but make progress to be qualified () at time . Due to the mapping between the decision rule and the selection rate, the impact of decisions on individuals’ future qualifications are captured by the impact of selection rates on the overall qualification profiles via some general functions and in model (7).

The goal of the decision maker is to find a decision rule with or without fairness consideration, so as to maximize . It examines what happens to the qualification profiles of two groups when these decisions are applied at each time, and under what conditions social equality is attained under these decisions.

Without fairness considerations, the corresponding optimal decision at time for , , is given by222Note that such an ideal decision rule assumes the knowledge of , which is not actually observable. In this sense this decision rule, which has 0 error, is not practically feasible. Our understanding is that the goal in Mouzannar et al. (2019) is to analyze what happens in such an ideal scenario when applying the perfect decision. :


Using this decision rule the selection rate . Since the decision rules for the two groups are not constrained by each other, the dynamics (7) can be simplified as follows: ,


Social equality can be attained for any starting profiles in this unconstrained case if and only if the system has a unique globally attracting equilibrium point and a sufficient condition is given in Mouzannar et al. (2019).

Mouzannar et al. (2019) also studied impact of fairness intervention on dynamics. It focuses on the notion of demographic parity (DP), which requires the selection rates of two groups to be equal, i.e., . Depending on group proportions and utilities , , there are two possibilities for the fair optimal decision rule . If group labels are assigned such that is the advantaged group, i.e., , then we have:

(under-selected) (11)