Exposing the Probabilistic Causal Structure of Discrimination

by   Francesco Bonchi, et al.
NYU college
ISI Foundation

Discrimination discovery from data is an important task aiming at identifying patterns of illegal and unethical discriminatory activities against protected-by-law groups, e.g., ethnic minorities. While any legally-valid proof of discrimination requires evidence of causality, the state-of-the-art methods are essentially correlation-based, albeit, as it is well known, correlation does not imply causation. In this paper we take a principled causal approach to the data mining problem of discrimination detection in databases. Following Suppes' probabilistic causation theory, we define a method to extract, from a dataset of historical decision records, the causal structures existing among the attributes in the data. The result is a type of constrained Bayesian network, which we dub Suppes-Bayes Causal Network (SBCN). Next, we develop a toolkit of methods based on random walks on top of the SBCN, addressing different anti-discrimination legal concepts, such as direct and indirect discrimination, group and individual discrimination, genuine requirement, and favoritism. Our experiments on real-world datasets confirm the inferential power of our approach in all these different tasks.



There are no comments yet.


page 1

page 2

page 3

page 4


Marrying Fairness and Explainability in Supervised Learning

Machine learning algorithms that aid human decision-making may inadverte...

Supervised learning algorithms resilient to discriminatory data perturbations

The actions of individuals can be discriminatory with respect to certain...

On Discrimination Discovery and Removal in Ranked Data using Causal Graph

Predictive models learned from historical data are widely used to help c...

Audits as Evidence: Experiments, Ensembles, and Enforcement

We develop tools for utilizing correspondence experiments to detect ille...

Probabilistic Causal Analysis of Social Influence

Mastering the dynamics of social influence requires separating, in a dat...

A Causal Approach for Unfair Edge Prioritization and Discrimination Removal

In budget-constrained settings aimed at mitigating unfairness, like law ...

Achieving non-discrimination in prediction

Discrimination-aware classification is receiving an increasing attention...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

The importance of discrimination discovery. At the beginning of 2014, as an answer to the growing concerns about the role played by data mining algorithms in decision-making, USA President Obama called for a 90-day review of big data collecting and analysing practices. The resulting report111http://www.whitehouse.gov/sites/default/files/docs/big_data_privacy_report_may_1_2014.pdf concluded that “big data technologies can cause societal harms beyond damages to privacy". In particular, it expressed concerns about the possibility that decisions informed by big data could have discriminatory effects, even in the absence of discriminatory intent, further imposing less favorable treatment to already disadvantaged groups.

Discrimination refers to an unjustified distinction of individuals based on their membership, or perceived membership, in a certain group or category. Human rights laws prohibit discrimination on several grounds, such as gender, age, marital status, sexual orientation, race, religion or belief, membership in a national minority, disability or illness. Anti-discrimination authorities (such as equality enforcement bodies, regulation boards, consumer advisory councils) monitor, provide advice, and report on discrimination compliances based on investigations and inquiries. A fundamental role in this context is played by discrimination discovery in databases, i.e., the data mining problem of unveiling discriminatory practices by analyzing a dataset of historical decision records.

Discrimination is causal. According to current legislation, discrimination occurs when a group is treated “less favorably” [3] than others, or when “a higher proportion of people not in the group is able to comply” with a qualifying criterion [4]. Although these definitions do not directly imply causation, as stated in [5] all discrimination claims require plaintiffs to demonstrate a causal connection between the challenged outcome and a protected status characteristic. In other words, in order to prove discrimination, authorities must answer the counterfactual question: what would have happened to a member of a specific group (e.g., nonwhite), if he or she had been part of another group (e.g., white)?

“The Sneetches”, the popular satiric tale222http://en.wikipedia.org/wiki/The_Sneetches_and_Other_Stories against discrimination published in 1961 by Dr. Seuss, describes a society of yellow creatures divided in two races: the ones with a green star on their bellies, and the ones without. The Star-Belly Sneetches have some privileges that are instead denied to Plain-Belly Sneetches. There are, however, Star-On and Star-Off machines that can make a Plain-Belly into a Star-Belly, and viceversa. Thanks to these machines, the causal relationship between race and privileges can be clearly measured, because stars can be placed on or removed from any belly, and multiple outcomes can be observed for an individual. Therefore, we could readily answer the counterfactual question, saying with certainty what would have happened to a Plain-Belly Sneetch had he or she been a Star-Belly Sneetch.

In the real world however, proving discrimination episodes is much harder, as we cannot manipulate race, gender, or sexual orientation of an individual. This highlights the need to assess discrimination as a causal inference problem [6] from a database of past decisions, where causality can be inferred probabilistically. Unfortunately, the state of the art of data mining methods for discrimination discovery in databases does not properly address the causal question, as it is mainly based on correlation-based methods (surveyed in Section 2).

Correlation is not causation. It is well known that correlation between two variables does not necessarily imply that one causes the other. Consider a unique cause of two effects, and : if we do not take into account , we might derive wrong conclusions because of the observable correlation between and . In this situation, is said to act as a confounding factor for the relationship between and .

Variants of the complex relation just discussed can arise even if, in the example, is not the actual cause of either or , but it is only correlated to them, for instance, because of how the data were collected. Consider for instance a credit dataset where there exists high correlation between a variable representing low income and another variable representing loan denial and let us assume that this is due to an actual legitimate causal relationship in the sense that, legitimately, a loan is denied if the applicant has low income. Let us now assume that we also observe high correlation between low income and being female, which, for instance, can be due to the fact that the women represented in the specific dataset in analysis, tend to be underpaid. Given these settings, in the data we would also observe high correlation between the variable gender being female and the variable representing loan denial, due to the fact that we do not account for the presence of the variable low income. Following common terminologies, we will say that such situations are due to spurious correlations.

However, the picture is even more complicated: it could be the case, in fact, that being female is the actual cause of the low income and, hence, be the indirect cause of loan denial through low income. This would represent a causal relationship between the gender and the loan denial, that we would like to detect as discrimination.

Disentangling these two different cases, i.e., female is only correlated to low income in a spurious way, or being female is the actual cause of low income, is at the same time important and challenging. This highlights the need for a principled causal approach to discrimination detection.

Another typical pitfall of correlation-based reasoning is expressed by what is known as Simpson’s paradox333http://en.wikipedia.org/wiki/Simpson’s_paradox according to which, correlations observed in different groups might disappear when these heterogeneous groups are aggregated, leading to false positives cases of discrimination discovery. One of the most famous false-positive examples due to Simpson’s paradox occurred when in 1973 the University of California, Berkeley was sued for discrimination against women who had applied for admission to graduate schools. In fact, by looking at the admissions of 1973, it first appeared that men applying were significantly more likely to be admitted than women. But later, by examining the individual departments carefully, it was discovered that none of them was significantly discriminating against women. On the contrary, most departments had exercised a small bias in favor of women. The apparent discrimination was due to the fact that women tended to apply to departments with lower rates of admission, while men tended to apply to departments with higher rates [7]. Later in Section 5.5 we will use the dataset from this episode to highlight the differences between correlation-based and causation-based methods.

Spurious correlations can also lead to false negatives (i.e., discrimination existing but not being detected) as is commonly seen in “reverse-discrimination”. The typical case is when authorities take affirmative actions, e.g., with compensatory quota systems, in order to protect a minority group from a potential discrimination. Such actions, while trying to erase the supposed discrimination (i.e., the spurious correlation), fail to address the real underlying causes for discrimination, potentially ending up denying individual members of a privileged group from access to their rightful shares of social goods. In the early 70’s, a case involving the University of California at Davis Medical School highlighted one such incident as the school’s admissions program reserved 16 of the 100 places in its entering class for “disadvantaged" applicants, thus unintentionally reducing the chances of admission for a qualified applicant.444http://en.wikipedia.org/wiki/Regents_of_the_University_of_California_v._Bakke

These are just few typical examples of the pitfalls of correlation-based reasoning in the discovery of discrimination. Later in Section 5.5 we show concrete examples from real-world datasets where correlation-based methods to discrimination discovery are not satisfactory.

Our proposal and contributions. In this paper we take a principled causal approach to the data mining problem of discrimination detection in databases. Following Suppes’ probabilistic causation theory [8, 9] we define a method to extract, from a dataset of historical decision records, the causal structures existing among the attributes in the data.

In particular, we define the Suppes-Bayes Causal Network (SBCN), i.e., a directed acyclic graph (dag) where we have a node representing a Bernulli variable of the type for each pair attribute-value present in the database. In this dag an arc represents the existence of a causal relation between and (i.e., causes ). Moreover, each arc is labeled with a score, representing the strength of the causal relation.


 is a constrained Bayesian network reconstructed by means of maximum likelihood estimation (MLE) from the given database, where we force the conditional probability distributions induced by the reconstructed graph to obey Suppes’ constraints: i.e.,

temporal priority and probability rising. Imposing Suppes’ temporal priority and probability raising we obtain what we call the prima facie causes graph [9], which might still contain spurious causes (false positives). In order to remove these spurious case we add a bias term to the likelihood score, favoring sparser causal networks: in practice we sparsify the prima facie causes graph by extracting a minimal set of edges which best explain the data. This regularization is done by means of the Bayesian Information Criterion (BIC) [10].

The obtained SBCN provides a clear summary, amenable to visualization, of the probabilistic causal structures found in the data. Such structures can be used to reason about different types of discrimination. In particular, we show how using several random-walk-based methods, where the next step in the walk is chosen proportionally to the edge weights, we can address different anti-discrimination legal concepts. Our experiments show that the measures of discrimination produced by our methods are very strong, almost binary, signals: our measures are very clearly separating the discrimination and the non-discrimination cases.

To the best of our knowledge this is the first proposal of discrimination detection in databases grounded in probabilistic causal theory.

Roadmap. The rest of the paper is organized as follows. Section 2 discusses the state of the art in discrimination detection in databases. In Section 3 we formally introduce the SBCN and we present the method for extracting such causal network from the input dataset. Once extracted our SBCN, in Section 4 we show how to exploit it for different concepts of discrimination detection, by means of random-walk methods. Finally Section 5 presents our experimental assessment and comparison with correlation-based methods on two real-world datasets.

2 Related work

Discrimination analysis is a multi-disciplinary problem, involving sociological causes, legal reasoning, economic models, statistical techniques [11, 12]. Some authors [13, 14] study how to prevent data mining from becoming itself a source of discrimination. In this paper instead we focus on the data mining problem of detecting discrimination in a dataset of historical decision records, and in the rest of this section we present the most related literature.

Pedreschi et al. [15, 16, 17] propose a technique based on extracting classification rules (inductive part) and ranking the rules according to some legally grounded measures of discrimination (deductive part). The result is a (possibly large) set of classification rules, providing local and overlapping niches of possible discrimination. This model only deals with group discrimination.

Luong et al. [18] exploit the idea of situation-testing [19] to detect individual discrimination. For each member of the protected group with a negative decision outcome, testers with similar characteristics (-nearest neighbors) are considered. If there are significantly different decision outcomes between the testers of the protected group and the testers of the unprotected group, the negative decision can be ascribed to discrimination.

Zliobaite et al. [20] focus on the concept of genuine requirement to detect that part of discrimination which may be explained by other, legally grounded, attributes. In [21] Dwork et al. address the problem of fair classification that achieves both group fairness, i.e., the proportion of members in a protected group receiving positive classification is identical to the proportion in the population as a whole, and individual fairness, i.e., similar individuals should be treated similarly.

The above approaches assume that the dataset under analysis contains attributes that denote protected groups (i.e., direct discrimination). This may not be the case when such attributes are not available, or not even collectable at a micro-data level as in the case of the loan applicant’s race. In these cases we talk about indirect discrimination discovery. Ruggieri et al. [22, 23] adopt a form of rule inference to cope with the indirect discovery of discrimination. The correlation information is called background knowledge, and is itself coded as an association rule.

Mancuhan and Clifton [24]

propose Bayesian networks as a tool for discrimination discovery. Bayesian networks consider the dependence between all the attributes and use these dependencies in estimating the joint probability distribution without any strong assumption, since a Bayesian network graphically represents a factorization of the joint distribution in terms of conditional probabilities encoded in the edges. Although Bayesian networks are often used to represent causal relationships, this needs not be the case, in fact a directed edge from two nodes of the network does not imply any causal relation between them. As an example, let us observe that the two graphs

and impose exactly the same conditional independence requirements and, hence, any Bayesian network would not be able to disentangle the direction of any causal relationship among these events.

Our work departs from this literature as:

  1. it is grounded in probabilistic causal theory instead of being based on correlation;

  2. it proposes a holistic approach able to deal with different types of discrimination in a single unifying framework, while the methods in the state of the art usually deal with one and only one specific type of discrimination;

  3. it is the first work to adopt graph theory and social network analysis concepts, such as random-walk-based centrality measures and community detection, for discrimination detection;

Our proposal has also lower computational cost than methods such as [15, 16, 17] which require to compute a potentially exponential number of association/classification rules.

3 Suppes-Bayes Causal Network

In order to study discrimination as a causal inference problem, we exploit the criteria defined in the theories of probabilistic causation [8]. In particular, we follow [9], where Suppes proposed the notion of prima facie causation that is at the core of probabilistic causation. Suppes’ definition is based on two pillars: any cause must happen before its effect (temporal priority) and it must raise the probability of observing the effect (probability raising).

Definition 1 (Probabilistic causation [9])

For any two events and , occurring respectively at times and , under the mild assumptions that , the event is called a prima facie cause of the event if it occurs before the effect and the cause raises the probability of the effect, i.e.

In the rest of this section we introduce our method to construct, from a given relational table , a type of causal Bayesian network constrained to satisfy the conditions dictated by Suppes’ theory, which we dub Suppes-Bayes Causal Network (SBCN).

In the literature many algorithms exist to carry out structural learning of general Bayesian networks and they usually fall into two families [25]. The first family, constraint based learning, explicitly tests for pairwise independence of variables conditioned on the power set of the rest of the variables in the network. These algorithms exploit structural conditions defined in various approaches to causality [8, 26, 27]. The second family, score based learning, constructs a network which maximizes the likelihood of the observed data with some regularization constraints to avoid overfitting. Several hybrid approaches have also been recently proposed [28, 29, 30].

Our framework can be considered a hybrid approach exploiting constrained maximum likelihood estimation (MLE) as follows: we first define all the possible causal relationship among the variables in by considering only the oriented edges between events that are consistent with Suppes’ notion of probabilistic causation and, subsequently, we perform the reconstruction of the SBCN by a score-based approach (using BIC), which considers only the valid edges.

We next present in details the whole learning process.

3.1 Suppes’ constraints

We start with an input relational table defined over a set of categorical attributes and samples. In case continuous numerical attributes exists in , we assume they have been discretized to become categorical. From , we derive , an binary matrix representing Bernoulli variables of the type , where an entry is if we have an observation for the specific variable and otherwise.

Temporal priority. The first constraint, temporal priority, cannot be simply checked in the data as we have no timing information for the events. In particular, in our context the events for which we want to reason about temporal priority are the Bernoulli variables .

The idea here is that, e.g., cannot be a cause of , because the time when the gender of an individual is determined is antecedent to that of when the income is determined. This intuition is implemented by simply letting the data analyst provide as input to our framework a partial temporal order for the attributes, which is then inherited from the Bernoulli variables 555Note that our learning technique requires the input order to be correct and complete in order to guarantee its convergence. Nevertheless, if this is not the case, it is still capable of providing valuable insights about the underlying causal model, although with the possibility of false positive or false negative causal claims..

Based on the input dataset and the partial order we produce the first graph where we have a node for each of the Bernoulli variables, so , and we have an arc whenever . This way we will immediately rule out causal relations that do not satisfy the temporal priority constraint.

Probability raising. Given the graph built as described above the next step requires to prune the arcs which do not satisfy the second constraint, probability raising, thus building , where . In particular we remove from each arc such that . The graph so obtained is called prima facie graph.

We recall that the probability raising condition is equivalent to constraining for positive statistical dependence [29]: in the prima facie graph we model all and only the positive correlated relations among the nodes already partially ordered by temporal priority, consistently with Suppes’ characterization of causality in terms of relevance.

Figure 1: One portion of the SBCN extracted from the Adult dataset. This subgraph corresponds to the community reported later in Table 3 (Section 5) extracted by a community detection algorithm.

3.2 Network simplification

Suppes’ conditions are necessary but not sufficient to evaluate causation [30]: especially when the sample size is small, the model may have false positives (spurious causes), even after constraining for Suppes’ temporal priority and probability raising criteria (which aim at removing false negatives). Consequently, although we expect all the statistically relevant causal relations to be modelled in , we also expect some spurious ones in it.

In our proposal, in place of other structural conditions used in various approaches to causality, (see e.g., [8, 26, 27]), we perform a network simplification (i.e., we sparsify the network by removing arcs) with a score based approach, specifically by relying on the Bayesian Information Criterion (BIC) as the regularized likelihood score [10].

We consider as inputs for this score the graph and the dataset . Given these, we select the set of arcs that maximizes the score:

In the equation, denotes the graph, denotes the data, denotes the number of samples, and denotes the number of parameters in . Thus, the regularization term favors graphs with fewer arcs. The coefficient weighs the regularization term, such that the higher the weight, the more sparsity will be favored over “explaining" the data through maximum likelihood. Note that the likelihood is implicitly weighted by the number of data points, since each point contributes to the score.

Assume that there is one true (but unknown) probability distribution that generates the observed data, which is, eventually, uniformly randomly corrupted by false positives and negatives rates (in ). Let us call correct model, the statistical model which best approximate this distribution. The use of BIC on

results in removing the false positives and, asymptotically (as the sample size increases), converges to the correct model. In particular, BIC is attempting to select the candidate model corresponding to the highest Bayesian Posterior probability, which can be proved to be equivalent to the presented score and its

penalization factor.

We denote with the graph that we obtain after this step. We note that, as for general Bayesian network, is a dag by construction.

3.3 Confidence score

Using the reconstructed SBCN, we can represent the probabilistic relationships between any set of events (nodes). As an example, suppose to consider the nodes representing respectively and being the only two direct causes (i.e., with arcs toward) of . Given SBCN, we can estimate the conditional probabilities for each node in the graph, i.e., probability of given in the example, by computing the conditional probability of only the pair of nodes directly connected by an arc. For an overview of state-of-the-art methods for doing this, see [25]. However, we expect to be mostly dealing with full data, i.e., for every directly connected node in the SBCN, we expect to have several observations of any possible combination . For this reason, we can simply estimate the node probabilities by counting the observations in the data. Moreover, we will exploit such conditional probabilities to define the confidence score of each arc in terms of their causal relationship.

In particular, for each arc involving the causal relationship between two nodes , we define a confidence score , which, intuitively, aims at estimating the observations where the cause is followed by its effect , that is , and the ones where this is not observed, i.e., , because of imperfect causal regularities. We also note that, by the constraints discussed above, we require and, for this reason, each weight is positive and no larger than 1, i.e., .

Combining all of the concepts discussed above, we conclude with the following definition.

Definition 2 (Suppes-Bayes Causal Network)

Given an input dataset of Bernoulli variables and samples, and given a partial order of the variables, the Suppes-Bayes Causal Network subsumed by is a weighted dag such that the following requirements hold:

  • [Suppes’ constraints] for each arc involving the causal relationship between nodes , under the mild assumptions that :

  • [Simplification] let be the set of arcs satisfying the Suppes’ constraints as before; among all the subsets of , the set of arcs is the one whose corresponding graph maximizes BIC:

  • [Score]

An example of a portion of a SBCN extracted from a real-world dataset is reported in Figure 1.

Algorithm 1 summarizes the learning approach adopted for the inference of the SBCN . Given an input dataset over Bernoulli variables and samples, and a partial order of the variables, Suppes’ constraints are verified (Lines -) to construct a dag as described in Section 3.1.

The likelihood fit is performed by hill climbing (Lines -), an iterative optimization technique that starts with an arbitrary solution to a problem (in our case an empty graph) and then attempts to find a better solution by incrementally visiting the neighbourhood of the current one. If the new candidate solution is better than the previous one it is considered in place of it. The procedure is repeated until the stopping criterion is matched.

The occurs (Line ) in two situations: the procedure stops when we have performed a large enough number of iterations or, it stops when none of the solutions in is better than the current . Note that denotes all the solutions that are derivable from by removing or adding at most one edge.

1:  Inputs: an input dataset of Bernoulli variables and samples, and a partial order of the variables
2:  Output: as in Definition 2
3:  [Suppes’ constraints]
4:  for all pairs among the Bernoulli variables do
5:     if  and  then
6:        add the arc to .
7:     end if
8:  end for
9:  [Likelihood fit by hill climbing]
10:  Consider .
11:  while  do
12:     Let be the neighbor solutions of .
13:     Remove from any solution whose arcs are not included in .
14:     Consider a random solution in .
15:     if  then
16:        .
17:         arc of , .
18:     end if
19:  end while
20:  .
21:  return  .
Algorithm 1 Learning the SBCN 

Time and space complexity. The computation of the valid dag according to Suppes’ constraints (Lines -) requires a pairwise calculation among the Bernoulli variables. After that, the likelihood fit by hill climbing (Lines -

) is performed. Being an heuristic, the computational cost of hill climbing depends on the sopping criterion. However, constraining by Suppes’ criteria tends to regularize the problem leading on average to a quick convergence to a good solution. The time complexity of Algorithm 

1 is and the space required is , where however is usually not too large, being the number of attribute-value pairs, and not the number of examples.

3.4 Expressivity of a SBCN

We conclude this Section with a discussion on the causal relations that we model by a .

Let us assume that there is one true (but unknown) probability distribution that generates the observed data whose structure can be modelled by a dag. Furthermore, let us consider the causal structure of such a dag and let us also assume each node with more then one cause to have conjunctive parents: any observation of the child node is preceded by the occurrence of all its parents. As before we call correct model, the statistical model which best approximate the distribution. On these settings, we can prove the following theorem.

Theorem 1

Let the sample size , the provided partial temporal order be correct and complete and the data be uniformly randomly corrupted by false positives and negatives rates (in ), then the inferred from the data is the correct model.

[Sketch] Let us first consider the case where the observed data have no noise. On such an input, we observe that the prima facie graph has no false negatives: in fact modelling a genuine causal relation, , thus the probability raising constraint is satisfied, so it is the temporal priority given that we assumed to be correct and complete.

Furthermore, it is know that the likelihood fit performed by converges to a class of structures equivalent in terms of likelihood among which there is the correct model: all these topologies are the same unless the directionality of some edges. But, since we started with the prima facie graph which is already ordered by temporal priority, we can conclude that in this case the coincides with the correct model.

To extend the proof to the case of data uniformly randomly corrupted by false positives and negatives rates (in ), we note that the marginal and joint probabilities change monotonically as a consequence of the assumption that the noise is uniform. Thus, all inequalities used in the preceding proof still hold, which concludes the proof.

In the more general case of causal topologies where any cause of a common effect is independent from any other cause (i.e., we relax the assumption of conjunctive parents), the is not guaranteed to converge to the correct model but it coincides with a subset of it modeling all the edges representing statistically relevant causal relations (i.e., where the probability raising condition is verified).

4 Discrimination discovery by random walks

In this section we propose several random-walk-based methods over the reconstructed SBCN, to deal with different discrimination-detection tasks.

4.1 Group discrimination and favoritism

The basic problem in the analysis of direct discrimination is precisely to quantify the degree of discrimination suffered by a given protected group (e.g., an ethnic group) with respect to a decision (e.g., loan denial). In contrast to discrimination, favoritism refers to the case of an individual treated better than others for reasons not related to individual merit or business necessity: for instance, favoritism in the workplace might result in a person being promoted faster than others unfairly. In the following we denote favoritism as positive discrimination in contrast with negative discrimination.

Given an SBCN we define a measure of group discrimination (either negative or positive) for each node . Recall that each node represents a pair , so it is essentially what we refer to as a group, e.g., . Our task is to assign a score of discrimination to each node, so that the closer is to 1 the more discriminated is the group represented by .

We compute this score by means of a number of random walks that start from and reaches either the node representing the positive decision or the one representing the negative decision. In these random walks the next step is chosen proportionally to the weights of the out-going arcs. Suppose a random walk has reached a node , and let denote the set of outgoing arcs from . Then the arc is chosen with probability

When a random walk ends in a node with no outgoing arc before reaching either the negative or the positive decision, it is restarted from the source node .

Definition 3 (Group discrimination score)

Given an , let and denote the nodes indicating the negative and positive decision, respectively. Given a node , and a number of random walks to be performed, we denote as the number of random walks started at node that reach earlier than . The discrimination score for the group corresponding to node is then defined as

This implicitly also defines a score of positive discrimination (or favoritism): .

Taking advantage of the SBCN  we also propose two additional measures capturing how far a node representing a group is from the positive and negative decision respectively. This is done by computing the average number of steps that the random walks take to reach the two decisions: we denote these scores as and .

4.2 Indirect discrimination

The European Union Legislation [4] provides a broad definition of indirect discrimination as occurring “where an apparently neutral provision, criterion or practice would put persons of a racial or ethnic origin at a particular disadvantage compared with other persons". In other words, the actual result of the apparently neutral provision is the same as an explicitly discriminatory one. A typical legal case study of indirect discrimination is concerned with redlining: e.g., denying a loan because of ZIP code, which in some areas is an attribute highly correlated to race. Therefore, even if the attribute race cannot be required at loan-application time (thus would not be present in the data), still race discrimination is perpetrated. Indirect discrimination discovery refers to the data mining task of discovering the attributes values that can act as a proxy to the protected groups and lead to discriminatory decisions indirectly [15, 16, 13].

In our setting, indirect discrimination can be detected by applying the same method described in Section 4.1.

4.3 Genuine requirement

The legal concept of genuine requirement refers to detecting that part of the discrimination which may be explained by other, legally-grounded, attributes; e.g., denying credit to women may be explainable by the fact that most of them have low salary or delay in returning previous credits. A typical example in the literature is the one of the “genuine occupational requirement", also called “business necessity" in [31, 32]. In the state of the art of data mining methods for discrimination discovery, it is also known as explainable discrimination [33] and conditional discrimination [20].

The task here is to evaluate to which extent the discrimination apparent for a group is “explainable” on a legal ground. Let be the node representing the group which is suspected of being discriminated, and be a node whose causal relation with a negative or positive decision is legally grounded. As before, and denote the negative and positive decision, respectively. Following the same random-walk process described in Section 4.1, we define the fraction of explainable discrimination for the group :

i.e., the fraction of random walks passing trough among the ones started in and reaching earlier than . Similarly we define , i.e., the fraction of explainable positive discrimination.

4.4 Individual and subgroup discrimination

Individual discrimination requires to measure the amount of discrimination for a specific individual, i.e., an entire record in the database. Similarly, subgroup discrimination refers to discrimination against a subgroup described by a combination of multiple protected and non-protected attributes: personal data, demographics, social, economic and cultural indicators, etc. For example, consider the case of gender discrimination in credit approval: although an analyst may observe that no discrimination occurs in general, it may turn out that older women obtain car loans only rarely.

Both problems can be handled by generalizing the technique introduced in Section 4.1 to deal with a set of starting nodes, instead of only one. Given an let be the nodes of interest. In order to define a discrimination score for , we perform a personalized PageRank[34] computation with respect to

. In personalized PageRank, the probability of jumping to a node when abandoning the random walk is not uniform, but it is given by a vector of probabilities for each node. In our case the vector will have the value

for each of the nodes and zero for all the others. The output of personalized PageRank is a score of proximity/relevance to for each other node in the network. In particular, we are interested in the score of the nodes representing the negative and positive decision: i.e., and respectively.

Definition 4 (Generalized discrimination score)

Given an , let and denote the nodes indicating the negative and positive decision, respectively. Given a set of nodes , we define the generalized (negative) discrimination score for the subgroup or the individual represented by as

This implicitly also defines a generalized score of positive discrimination: .

5 Experimental Evaluation

This section reports the experimental evaluation of our approach on four datasets, Adult, German credit and census-income

from the UCI Repository of Machine Learning Databases

666http://archive.ics.uci.edu/ml, and Berkeley Admissions Data from [36]. These are well-known real-life datasets typically used in discrimination-detection literature.

Adult: consists of 48,842 tuples and 10 attributes, where each tuple correspond to an individual and it is described by personal attributes such as age, race, sex, relationship, education, employment, etc. Following the literature, in order to define the decision attribute we use the income levels, 50K (negative decision) or 50K (positive decision). We use four levels in the partial order for temporal priority: age, race, sex, and native country are defined in the first level; education, marital status, and relationship are defined in the second level; occupation and work class are defined in the third class, and the decision attribute (derived from income) is the last level.

German credit: consists of 1000 tuples with 21 attributes on bank account holders applying for credit. The decision attribute is based on repayment history, i.e., whether the customer is labeled with good or bad credit risk. Also for this dataset the partial order for temporal priority has four orders. Personal attributes such as gender, age, foreign worker are defined in the first level. Personal attributes such as employment status and job status are defined in the second level. Personal properties such as savings status and credit history are defined in the third level, and finally the decision attribute is the last level.

Census-income: consists of 299,285 tuples and 40 attributes, where each tuple correspond to an individual and it is described by demographic and employment attributes such as age, sex, relationship, education, employment, ext. Similar to Adult dataset, the decision attribute is the income levels and we define four levels in the partial order for temporal priority.

Building the SBCN just take a handful of seconds in German credit and Adult, and few minutes in Census-income on a commodity laptop. The main characteristics of the extracted SBCN are reported in Table 1, while the distribution of the edges scores is plotted in Figure 2.

Dataset avgDeg maxInDeg maxOutDeg
Adult 92 230 2.5 7 19
German credit 73 102 1.39 3 7
Census-income 386 1426 3.69 8 54
Table 1: SBCN main characteristics.
Figure 2: Distribution of edge scores.

As discussed in the Introduction we also use the dataset from the famous 1973 episode at University of California at Berkeley, in order to highlight the differences between correlation-based and causation-based methods.

Berkeley Admissions Data: consists of 4,486 tuples and three attributes, where each tuple correspond to an individual and it is described by the gender of applicants and the department that they apply for it. For this dataset the partial order for temporal priority has three orders. Gender is defined in the first level, department in the second level, and finally the decision attribute is the last level. Table 2 is a three-way table that presents admissions data at the University of California, Berkeley in 1973 according to the variables department (A, B, C, D, E), gender (male, female), and outcome (admitted, denied). The table is adapted from data in the text by Freedman, et al. [36].

Male Female
Admitted Denied Admitted Denied Department
512 313 89 19 A
313 207 17 8 B
120 205 202 391 C
138 279 131 244 D
53 138 94 299 E
22 351 24 317 F
Table 2: Berkeley Admission Data

5.1 Community detection on the Sbcn

Given that our SBCN is a directed graph with edge weight, as a first characterization we try to partition it using a random-walks-based community detection algorithm, called Walktrap and proposed in [35], whose unique parameter is the maximum number of steps in a random walk (we set it to 8), and which automatically identifies the right number of communities. The idea is that short random walks tend to stay in the same community (densely connected area of the graph). Using this algorithm over the reconstructed SBCN from Adult dataset, we obtain communities: two larger ones and three smaller ones (reported in Table 3). Interestingly, the two larger communities seem built around the negative () and the positive () decisions.

Figure 1 in Section 3 shows the subgraph of the SBCN corresponding to (that we can call, the favoritism cluster): we note that such cluster also contains nodes such as sex_Male, age_old, relationship_Husband. The other large community , can be considered the discrimination cluster: beside the negative decision it contains other nodes representing disadvantaged groups such as sex_Female, age_young, race_Black, marital_status_Never_married. This good separability of the SBCN in the two main clusters of discrimination and favoritism, highlights the goodness of the causal structure captured by the SBCN.

negative_dec, wc:Private, ed:Some_college, ed:Assoc_acdm,
ms:Never_married, ms:Divorced, ms:Widowed,
ms:Married_AF_spouse, oc:Sales, oc:Other_service,
oc:Priv_house_serv, re:Own_child, re:Not_in_family, re:Wife,
re:Unmarried, re:Other_relative, ra:Black, oc:Armed_Forces,
oc:Handlers_cleaners, oc:Tech_support, oc:Transport_moving,
ed:7th_8th, ed:10th, ed:12th, ms:Separated,
ed:HS_grad,ed:11th, nc:Outlying_US_Guam_USVI_etc,
nc:Haiti, ag:young, sx:Female, ra:Amer_Indian_Eskimo,
nc:Trinadad_Tobago, nc:Jamaica, oc:Machine_op_inspct,
ms:Married_spouse_absent, oc:Adm_clerical,
positive_dec, oc:Prof_specialty, wc:Self_emp_not_inc,
ms:Married_civ_spouse, oc:Craft_repair,oc:Protective_serv,
re:Husband, ed:Prof_school, wc:Self_emp_inc,
ag:old , wc:Local_gov, oc:Exec_managerial,
ed:Bachelors, ed:Assoc_voc, ed:Masters, wc:Never_worked,
wc:State_gov, ed:Doctorate, sx:Male, nc:India, nc:Cuba
oc:Farming_fishing, wc:Without_pay, nc:Mexico, nc:Canada,
nc:Italy, nc:Guatemala, nc:El_Salvador, ra:White,
nc:Poland, ed:1st_4th, ed:9th,ed:Preschool, ed:5th_6th
nc:Iran, nc:Puerto_Rico, nc:Dominican_Republic,
nc:Columbia, nc:Peru, nc:Nicaragua, ra:Other
nc:Philippines, nc:Cambodia, nc:China, nc:South,
nc:Japan, nc:Taiwan, nc:Hong, nc:Laos, nc:Thailand,
nc:Vietnam, ra:Asian_Pac_Islander
Table 3: Communities found in the SBCN extracted from the Adult dataset by Walktrap[35]. In the table the attributes are shortened as in parenthesis: age (ag), education (ed), marital_status (ms), native_country (nc), occupation (oc), race(ra), relationship (re), sex (sx), workclass (wc).

5.2 Group discrimination and favoritism

We next focus on assessing the discrimination score we defined in Section 4.1, as well as the average number of steps that the random walks take to reach the negative and positive decisions, denote and respectively.

Tables 6, 6 and 6 report the top-5 and bottom-5 nodes w.r.t. the discrimination score , for datasets Adult, German and Census-income, respectively. The first and most important observation is that our discrimination score provides a very clear signal, with some disadvantaged groups having very high discrimination score (equal to 1 or very close), and similarly clear signals of favoritism, with groups having , or equivalently . This is more clear in the Adult dataset, where the positive and negative decisions are artificially derived from the income attribute. In the German credit dataset, which is more realistic as the decision attribute is truly about credit, both discrimination and favoritism are less palpable. This is also due to the fact that German credit contains less proper causal relations, as reflected in the higher sparsity of the SBCN. A consequence of this sparsity is also that the random walks generally need more steps to reach one of the two decisions. In Census-income dataset, we observe favoritism with respect to married and asian_pacific individuals.

5.3 Genuine requirement

We next focus on genuine requirement (or explainable discrimination). Table 7 reports some examples of fraction of explainable discrimination (both positive and negative) on the Adult dataset. We can see how some fractions of discrimination against protected groups, can be “explained” by intermediate nodes such as having a low education profile, or a simple job. In the case these intermediate nodes are considered legally grounded, then one cannot easily file a discrimination claim.

Similarly, we can observe that the favoritism towards groups such as married men, is explainable, to a large extent, by higher education and good working position, such as managerial or executive roles.

relationship_Unmarried 1 1.164 -
marital_status_Never_married 0.996 1.21 2.14
age_Young 0.995 2.407 3.857
race_Black 0.994 2.46 4.4
sex_Female 0.98 2.60 3.76
relationship_Husband 0 - 2
marital_status_Married_civ_spouse 0 - 2.06
sex_Male 0 - 3.002
native_country_India 0.002 4.0 3.25
age_Old 0.018 2.062 2.14

Table 5: Top-5 and bottom-4 groups by discrimination score in German credit. We report only the bottom-4, because there are only 4 nodes in which .
residence_since_le_1d6 1 6.0 -
residence_since_gt_2d8 1 2.23 -
residence_since_from_1d6_le_2d2 1 6.0 -
age_gt_52d6 0.86 3.68 4.0
personal_status_male_single 0.791 5.15 5.0
job_unskilled_resident 0 - 2.39
personal_status_male_mar_or_wid 0.12 8.0 4.4
age_le_30d2 0.186 7.0 3.34
personal_status_female_ 0.294 6.48 4.4

Table 6: Top-5 and bottom-5 groups by discrimination score in Census-income dataset.
MIGSAME_Not_in_universe_under_1_year_old 0.71 4.09 8.82
WKSWORK_94_5_inf 0.625 3.0 6.76
AWKSTAT_Not_in_labor_force 0.59 2.0 6.16
VETYN_0_5_20_5 0.58 1.01 5.17
MARSUPWT_3188_455_4277_98 0.55 5.0 9.25
AHGA_Doctorate_degreePhD_EdD 0 - 3.07
AMARITL_Married_A_F_spouse_present 0 - 4.49
AMJOCC_Sales 0 - 2.0
ARACE_Asian_or_Pacific_Islander 0 - 6.47
VETYN_20_5_32_5 0 - 5.89
Table 4: Top-5 and bottom-5 groups by discrimination score in Adult dataset.
Source node Intermediate
race_Amer_Indian_Eskimo education_HS_grad
sex_Female occupation_Other_service
age_Young occupation_Other_service
relationship_Unmarried education_HS_grad
race_Black education_11th

Source node Intermediate
relationship_Husband occupation_Exec_managerial
sex_Male occupation_Exec_managerial
native_country_Iran education_Bachelors
native_country_India education_Prof_school
age_Old occupation_Exec_managerial
Table 7: Fraction of explainable discrimination for some exemplar pair of nodes in the Adult dataset.

5.4 Subgroup and Individual Discrimination

We next turn our attention to subgroup and individual discrimination discovery. Here the problem is to assign a score of discrimination not to a single node (a group), but to multiple nodes (representing the attributes of an individual or a subgroup of citizens). In Section 4.4 we have introduced based on the PageRank of the positive and negative decision, and respectively, personalized on the nodes of interest. Figure 3 presents a scatter plot of versus for each individual in the German credit dataset. We can observe the perfect separation between individuals corresponding to a high personalized PageRank with respect to the positive decision, and those associated with a high personalized PageRank relative to the negative decision.

Figure 3: Scatter plot of versus for each individual in the German credit dataset.

Such good separation is also reflected in the generalized discrimination score (Definition 4) that we obtain by combining versus .

In Figure 4 we report the distribution of the generalized discrimination score for the population of the German credit dataset: we can make a note of the clear separation between the two subgroups of the population.

Figure 4: Individual discrimination: histogram representing the distribution of the values of the generalized discrimination score for the population of the German credit dataset.

In the Adult dataset (Figure 5) we do not observe the same neat separation in two subgroups as in the German credit dataset, also due to the much larger number of points. Nevertheless, as expected, and still exhibit anticorrelation. In Figure 5 we also use colors to show two different groups: red dots are for age_Young and blue dots are for age_Old individuals. As expected we can see that the red dots are distributed more in the area of higher .

Figure 5: Individual discrimination: scatter plot of versus for each individual in the Adult dataset. Red dots are for age_Young and blue dots are for age_Old.

The plots in Figure 6 have a threshold on the X-axis, and the fraction of tuples having on the Y-axis, and they show this for different subgroups. The first plot, from the Adult dataset, shows the group female, young, and young female. As we can see the individuals that are both young and female have a higher generalized discrimination score. Similarly, the second plot shows the groups old, single male, and old single male from the German credit dataset. Here we can observe much lower rates of discrimination with only of the corresponding populations having , while in the previous plot it was more than 85%.

Figure 6: Subgroup discrimination: plots reporting a threshold on the X-axis, and the fraction of tuples having on the Y-axis. The top plot is from Adult, while the bottom is from German credit.

5.5 Comparison with prior art

In this section, we discuss examples in which our causation-based method draws different conclusions from the correlation-based methods presented in [15, 16, 17] using the same datasets and the same protected groups 777We could not compare with [24] due to repeatability issues..

- +
Figure 7: Contingency table for foreign_worker in the German credit dataset.
- +
Figure 8: Contingency table for race_black in the Adult dataset.

The first example involves the foreign_worker group from German Credit dataset, whose contingency table is reported in Figure 8. Following the approaches of [15, 16, 17] the foreign_worker group results strongly discriminated. In fact Figure 8 shows an value (risk difference) of 0.244 which is considered a strong signal: in fact is already considered discrimination [17].

However, we can observe that the foreign_worker group is per se not very significant, as it contains 963 tuples out of 1000 total. In fact our causal approach does not detect any discrimination with respect to foreign_worker which appears as a disconnected node in the SBCN.

The second example is in the opposite direction. Consider the race_black group from Adult dataset whose contingency table is shown in Figure 8. Our causality-based approach detects a very strong signal of discrimination (), while the approaches of [15, 16, 17] fail to discover discrimination against black minority when the value of minimum support threshold used for extracting classification rules is more than . On the other hand, when such minimum support threshold is kept lower, the number of extracted rules might be overwhelming.

Finally, we turn our attention to the famous example of false-positive discrimination case happened at Berkeley in 1973, that we discussed in Section 1. Figure 9 presents the SBCN extracted by our approach from Berkeley Admission Data. Interestingly, we observe that there is no direct edge between node sex_Female and Admission_No. And sex_Female is connected to node Admission_No through nodes of Dep_C, Dep_D, Dep_E, and Dep_F, which are exactly the departments that have lower admission rate. By running our random walk-based methods over SBCN  we obtain the value of for the score of explainable discrimination confirming that apparent discrimination in this dataset is due the fact that women tended to apply to departments with lower rates of admission.

Figure 9: The SBCN constructed from Berkeley Admission Data dataset.

Similarly, we observe that there is no direct edge between node sex_Male and Admission_Yes. And sex_Male is connected to node Admission_Yes through nodes of Dep_A, and Dep_B, which are exactly the departments that have higher admission rate. By running our random walk-based methods over SBCN  we obtain the value of for the score of explainable discrimination confirming that apparent favoritism towards men is due to the fact that men tended to apply to departments with higher rates of admission.

- +
Figure 10: Contingency table for female in the Berkeley Admission Data dataset.

However, following the approaches of  [15, 16, 17], the contingency table shown in Figure 10 can be extracted from Berkeley Admission Data. As shown in Figure 10, the value of suggests a signal of discrimination versus women.

This highlights once more the pitfalls of correlation-based approaches to discrimination detection and the need for a principled causal approach, as the one we propose in this paper.

6 Conclusions

Discrimination discovery from databases is a fundamental task in understanding past and current trends of discrimination, in judicial dispute resolution in legal trials, in the validation of micro-data before they are publicly released. While discrimination is a causal phenomenon, and any discrimination claim requires to prove a causal relationship, the bulk of the literature on data mining methods for discrimination detection is based on correlation reasoning.

In this paper, we propose a new discrimination discovery approach that is able to deal with different types of discrimination in a single unifying framework. It is the first discrimination detection method grounded in probabilistic causal theory. We define a method to extract a graph representing the causal structures found in the database, and then we propose several random-walk-based methods over the causal structures, addressing a range of different discrimination problems.

Our experimental assessment confirmed the great flexibility of our proposal in tackling different aspects of the discrimination detection task, and doing so with very clean signals, clearly separating discrimination cases.

Repeatability. Our software together with the datasets used in the experiments are available at http://bit.ly/1GizSIG.


  • [1]
  • [2]
  • [3] Australian Legislation, “Equal opportunity act–victoria state,(b) anti-discrimination act–queensland state, 2008.”
  • [4] EU Legislation, “(a) race equality directive,2000/43/ec, 2000; (b) employment equality directive, 2000/78/ec, 2000; (c) equal treatment of persons, european parliament legislative resolution, 2009.”
  • [5] S. R. Foster, “Causation in antidiscrimination law: Beyond intent versus impact,” Hous. L. Rev., vol. 41, p. 1469, 2004.
  • [6] M. Dabady, R. M. Blank, C. F. Citro et al., Measuring racial discrimination.   National Academies Press, 2004.
  • [7] P. J. Bickel, E. A. Hammel, and J. W. O’Connell, “Sex bias in graduate admissions: Data from berkeley,” Science, vol. 187, no. 4175, 1975.
  • [8] C. Hitchcock, “Probabilistic causation,” in The Stanford Encyclopedia of Philosophy, 2012.
  • [9] P. Suppes, A probabilistic theory of causality.   North-Holland Publishing Company, 1970.
  • [10] G. Schwarz, “Estimating the dimension of a model,” Annals of Statistics, vol. 6, no. 2, pp. 461–464, 1978.
  • [11] B. Custers, T. Calders, B. Schermer, and T. Zarsky, Discrimination and Privacy in the Information Society: Studies in Applied Philosophy, Epistemology and Rational Ethics.   Springer, 2012, vol. 3.
  • [12] A. Romei and S. Ruggieri, “A multidisciplinary survey on discrimination analysis,”

    The Knowledge Engineering Review

    , pp. 1–57, 2013.
  • [13] S. Hajian and J. Domingo-Ferrer, “A methodology for direct and indirect discrimination prevention in data mining,” TKDE, 25(7), 2013.
  • [14] F. Kamiran and T. Calders, “Data preprocessing techniques for classification without discrimination,” KAIS, 33(1), 2012.
  • [15] D. Pedreshi, S. Ruggieri, and F. Turini, “Discrimination-aware data mining,” in KDD, 2008.
  • [16] D. Pedreschi, S. Ruggieri, and F. Turini, “Measuring discrimination in socially-sensitive decision records.” in SDM, 2009.
  • [17] S. Ruggieri, D. Pedreschi, and F. Turini, “Data mining for discrimination discovery,” TKDD, 4(2), 2010.
  • [18] B. T. Luong, S. Ruggieri, and F. Turini, “k-nn as an implementation of situation testing for discrimination discovery and prevention,” in KDD, 2011.
  • [19] I. Rorive, “Proving discrimination cases: The role of situation testing,” 2009.
  • [20] I. Zliobaite, F. Kamiran, and T. Calders, “Handling conditional discrimination,” in ICDM, 2011.
  • [21] C. Dwork, M. Hardt, T. Pitassi, O. Reingold, and R. Zemel, “Fairness through awareness,” in ITCS, 2012.
  • [22] D. Pedreschi, S. Ruggieri, and F. Turini, “Integrating induction and deduction for finding evidence of discrimination,” in ICAIL, 2009.
  • [23] S. Ruggieri, S. Hajian, F. Kamiran and X. Zhang, “Anti-discrimination Analysis Using Privacy Attack Strategies,” in PKDD, 2014.
  • [24] K. Mancuhan and C. Clifton, “Combating discrimination using bayesian networks,” Artificial Intelligence and Law, 22(2), 2014.
  • [25] D. Koller and N. Friedman, Probabilistic graphical models: principles and techniques.   MIT press, 2009.
  • [26] P. Menzies, “Counterfactual theories of causation,” in The Stanford Encyclopedia of Philosophy, spring 2014 ed., E. N. Zalta, Ed., 2014.
  • [27] J. Woodward, “Causation and manipulability,” in The Stanford Encyclopedia of Philosophy, winter 2013 ed., E. N. Zalta, Ed., 2013.
  • [28] E. Brenner and D. Sontag, “Sparsityboost: A new scoring function for learning bayesian network structure,” arXiv:1309.6820, 2013.
  • [29] L. O. Loohuis, G. Caravagna, A. Graudenzi, D. Ramazzotti, G. Mauri, M. Antoniotti, and B. Mishra, “Inferring tree causal models of cancer progression with probability raising,” PLoS ONE, 9(12), 2014.
  • [30] D. Ramazzotti et al. “Capri: Efficient inference of cancer progression models from cross-sectional data,” Bioinformatics, 31(18), 2015.
  • [31] US Federal Legislation, “(a) equal credit opportunity act, (b) fair housing act, (c) intentional employment discrimination, (d) equal pay act, (e) pregnancy discrimination act, (f) civil right act., 2010.”
  • [32] E. Ellis and P. Watson, EU anti-discrimination law.   Oxford University Press, 2012.
  • [33] S. Hajian, J. Domingo-Ferrer, A. Monreale, D. Pedreschi, and F. Giannotti, “Discrimination-and privacy-aware patterns,” DAMI, 29(6), 2015.
  • [34] G. Jeh and J. Widom, “Scaling personalized web search,” in WWW 2003.
  • [35] P. Pons and M. Latapy, “Computing communities in large networks using random walks,” in ISCIS, 2005.
  • [36] D. Feedman, R. Pisani, and R. Purves, “Statistics,” 3rd Edition, WW Norton & Company, New York (1998).