FairGNN: Eliminating the Discrimination in Graph Neural Networks with Limited Sensitive Attribute Information

09/03/2020 ∙ by Enyan Dai, et al. ∙ Penn State University 0

Graph neural networks (GNNs) have shown great power in modeling graph structured data. However, similar to other machine learning models, GNNs may make predictions biased on protected sensitive attributes, e.g., skin color, gender, and nationality. Because machine learning algorithms including GNNs are trained to faithfully reflect the distribution of the training data which often contains historical bias towards sensitive attributes. In addition, the discrimination in GNNs can be magnified by graph structures and the message-passing mechanism. As a result, the applications of GNNs in sensitive domains such as crime rate prediction would be largely limited. Though extensive studies of fair classification have been conducted on i.i.d data, methods to address the problem of discrimination on non-i.i.d data are rather limited. Furthermore, the practical scenario of sparse annotations in sensitive attributes is rarely considered in existing works. Therefore, we study the novel and important problem of learning fair GNNs with limited sensitive attribute information. FairGNN is proposed to eliminate the bias of GNNs whilst maintaining high node classification accuracy by leveraging graph structures and limited sensitive information. Our theoretical analysis shows that FairGNN can ensure the fairness of GNNs under mild conditions given limited nodes with known sensitive attributes. Extensive experiments on real-world datasets also demonstrate the effectiveness of FairGNN in debiasing and keeping high accuracy.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1. introduction

Graph neural networks (GNNs) (Bruna et al., 2013; Kipf and Welling, 2016; Veličković et al., 2017; Hamilton et al., 2017)

have achieved remarkable performance on various domains such as knowledge graph 

(Hamaguchi et al., 2017; Wang et al., 2018), social media mining (Hamilton et al., 2017)

, nature language processing 

(Kipf and Welling, 2016; Yao et al., 2019), and recommendation system (Ying et al., 2018; Berg et al., 2017). Generally, message-passing process is adopted in GNNs (Kipf and Welling, 2016; Hamilton et al., 2017), where information from neighbors is aggregated for every node in each layer. This process enriches node representations, and preserves both node feature characteristics and topological structures.

Despite the success in modeling graph data, GNNs trained on graphs may inherit the societal bias in data, which limits the adoption of GNNs in many real-world applications. First, extensive studies (Dwork et al., 2012; Beutel et al., 2017; Creager et al., 2019) have revealed that historical data may include patterns of previous discrimination and societal bias. Machine learning models trained on such data can inherit the bias on sensitive attributes such as ages, genders, skin color, and regions (Dwork et al., 2012; Beutel et al., 2017; Creager et al., 2019; Buolamwini and Gebru, 2018), which implies that GNNs could also exhibit the bias. Second, the topology of graphs and the message-passing of GNNs could magnify the bias. Generally, in graphs such as social networks, nodes of similar sensitive attributes are more likely to connect to each other than nodes of different sensitive attributes (Dong et al., 2016; Rahman et al., 2019; Mehrabi et al., 2019; Olteanu et al., 2019). For example, young people tend to build friendship with people of similar age on the social network (Dong et al., 2016). This makes the aggregation of neighbors’ features in GNN have similar representations for nodes of similar sensitive information while different representations for nodes of different sensitive features, leading to severe bias in decision making, i.e., the predictions are highly correlated with the sensitive attributes of the nodes. Our preliminary experiments in Sec. 3.5 indicate that GNNs have larger bias due to the adoption of graph structure than models which only use node attributes, which verifies our intuition. The bias would largely limit the wide adoption of GNNs in domains such as ranking of job applicants (Mehrabi et al., 2019), crime rate prediction (Suresh and Guttag, 2019)

, and credit score estimation 

(Zemel et al., 2013). Thus, it is critically important to investigate fair GNNs.

However, developing fair GNNs is a non-trivial task. First, to achieve fairness, we need to obtain abundant nodes with known sensitive attributes so that we can either revise the data or regularize the model; whereas people are unwilling to share their sensitive information in the real-world, and resulting in inadequate nodes with sensitive attributes known for fair model learning. For example, only 14% teen users public their complete profiles on Facebook (Madden et al., 2013). The lacking of sensitive information challenges many existing work on fair models (Beutel et al., 2017; Locatello et al., 2019; Louizos et al., 2015; Creager et al., 2019; Song et al., 2018; Liao et al., 2019; Madras et al., 2018; Zhang et al., 2018). Second, though extensive efforts have been made to establish fair models by revising features (Zhang et al., 2017; Kamiran and Calders, 2009, 2012), disentanglement (Louizos et al., 2015; Creager et al., 2019), adversarial debiasing (Edwards and Storkey, 2015; Beutel et al., 2017; Zhang et al., 2018; Madras et al., 2018) and fairness constraints (Zafar et al., 2015, 2017), they are overwhelmingly dedicated to independently and identically distributed (i.i.d) data, which cannot be directly applied on graph data for the absence of simultaneous consideration of the bias from node attributes and graph structures. Recently, (Rahman et al., 2019; Bose and Hamilton, 2019) aim to learn fair node representations from graphs. These methods merely deal with plain graphs without any node attributes, and focus on fair node representations instead of fair node classifications.

Therefore, in this paper, we study a novel problem of learning fair graph neural networks with limited sensitive information. In essence, we need to solve two challenges: (i) how to overcome the shortage of sensitive attributes for eliminating discrimination; and (ii) how to ensure the fairness of the GNN classifier. In an attempt to address these challenges, we propose a novel framework named as

FairGNN for fair node classification. A GNN sensitive attribute estimator is adopted in FairGNN to predict plenty of sensitive attributes with noise for fair classification. Inspired by existing works of fair classification on i.i.d data with adversarial learning (Edwards and Storkey, 2015; Beutel et al., 2017; Zhang et al., 2018; Madras et al., 2018), we deploy an adversary to ensure the GNN classifier make predictions independent with the estimated sensitive attributes. To further stabilize the training process and performance in fairness, we introduce a fairness constraint to make the predictions invariant with the estimated sensitive attributes. Our main contributions are:

  • We study a novel problem of fair graph neutral networks learning with limited sensitive information;

  • A new framework, FairGNN, is proposed to settle the shortage of sensitive attributes for adversarial debiasing and fairness constraint by estimating users’ sensitive attributes;

  • We conduct theoretical analysis showing fairness achieves at the global minimum even with estimated sensitive attributes;

  • Extensive experiments on different datasets demonstrate the effectiveness of our methods in eliminating discrimination while keeping high accuracy of GNNs.

The rest of the paper is organized as follows. In Sec. 2, we review related work. In Sec. 3, we conduct preliminary analysis to understand the bias issue of GNNs. In Sec. 4, we give the details of FairGNN. In Sec. 5, we conduct experiments to show the effectiveness of FairGNN. In Sec. 6, we conclude with future work.


2. related work

In this section, we will review related work including graph neural networks and fairness in machine learning.

2.1. Graph Neural Networks

Graph neural networks (GNNs), which generalize neural networks for graph structured data, have shown great success for various applications (Hamaguchi et al., 2017; Wang et al., 2018; Kipf and Welling, 2016; Yao et al., 2019; Ying et al., 2018; Berg et al., 2017). Generally, GNNs can be categorized into two categories, i.e., spectral-based (Bruna et al., 2013; Henaff et al., 2015; Defferrard et al., 2016; Kipf and Welling, 2016; Levie et al., 2018) and spatial-based (Veličković et al., 2017; Hamilton et al., 2017; Chen et al., 2018; Chiang et al., 2019; Ying et al., 2018). Spectral-based GNNs define graph convolution based on spectral graph theory, which is first explored by Bruna et al. (Bruna et al., 2013). Since then, more spectral-based methods are developed for further improvements and extensions (Henaff et al., 2015; Defferrard et al., 2016; Kipf and Welling, 2016; Levie et al., 2018). Graph Convolutional Network (GCN) (Kipf and Welling, 2016) is a particularly popular method which simplifies the convolutional operation on the graph. Spatial-based graph convolution directly updates the node representation by aggregating its neighborhoods’ representations (Niepert et al., 2016; Gilmer et al., 2017; Hamilton et al., 2017; Ying et al., 2018). Veličković et al. (Veličković et al., 2017) introduce the self-attention into the aggregation of spatial graph convolution by assigning higher weights to the more important nodes in graph attention network (GAT). Various spatial methods are proposed to solve the scalability issue of GCN (Hamilton et al., 2017; Chen et al., 2018; Chiang et al., 2019). For example, a neighbor sampling method to make it possible to train GNN with nodes in mini-batch instead of the whole graph is developed in GraphSAGE (Hamilton et al., 2017). Moreover, spatial-based methods have already been successfully deployed to deal with extremely large industrial datasets (Ying et al., 2018).

The essential idea of GNNs is to propagate the information of nodes through the graph to get better representations. However, people tend to build relationship with those sharing the same sensitive attributes. Then, representations in GNNs are nearly propagated within the subgroup, which highly increases the risk of discrimination towards sensitive attributes. Despite the risk of discrimination in GNNs, there is no existing work to address this problem. Thus, we study the novel problem of learning fair GNNs which will eliminate the potential discrimination.

2.2. Fairness in Machine Learning

Many works have been conducted to deal with the bias in the training data to achieve fairness in machine learning (Zhang et al., 2017; Kamiran and Calders, 2009, 2012; Beutel et al., 2017; Locatello et al., 2019; Dwork et al., 2012; Hardt et al., 2016). Based on which stage of the machine learning training process is revised, algorithms could be split into three categories: the pre-processing approaches, the in-processing approaches, and the post-processing approaches. The pre-processing approaches are applied before training machine learning models. They could reduce the bias by modifying the training data through correcting labels (Zhang et al., 2017; Kamiran and Calders, 2009), revising attributes of data (Kamiran and Calders, 2012; Feldman et al., 2015), generating non-discriminatory labeled data (Xu et al., 2018, 2019; Sattigeri et al., 2019), and obtaining fair data representations (Beutel et al., 2017; Locatello et al., 2019; Edwards and Storkey, 2015; Zemel et al., 2013; Louizos et al., 2015; Creager et al., 2019; Song et al., 2018; Liao et al., 2019; Madras et al., 2018). The in-processing approaches are designed to revise the training of the state-of-the-art models. Typically the machine learning models are trained with additional regularization terms or a new objective function. (Dwork et al., 2012; Zafar et al., 2015; Kamishima et al., 2011; Zhang et al., 2018). Finally, the post-processing approaches directly change the predictive labels to ensure fairness (Hardt et al., 2016; Pleiss et al., 2017). Recently, several works explore the learning of fair graph embeddings for recommendation (Rahman et al., 2019; Bose and Hamilton, 2019). Fairwalk (Rahman et al., 2019) modifies the random walk procedure of node2vec (Grover and Leskovec, 2016) to obtain a more diverse network neighborhood representations. The sensitive attributes of all the nodes are required in the sampling procedure of FairWalk. Bose and Hamilton (Bose and Hamilton, 2019) propose to add discriminators to eliminate the sensitive information in the graph embeddings. Similar to Fairwalk, the training process of the discriminators is in need of the sensitive attributes of all the nodes.

Our work is inherently different from existing works: (i) we focus on learning fair GNNs for node classification instead of fair graph embeddings; (ii) we address the problem that only a limited number of nodes are provided with sensitive attributes in practice.

3. Preliminaries Analysis

In this section, we first conduct preliminary analysis on real-world datasets to show that GNNs could exhibit more serve bias due to the graph structure and the message-passing. Sequentially, We formally give the problem definition of fair node classification.

3.1. Notations

We use to denote an attributed graph, where is the set of nodes, is the set of edges, and is the set of node features. is the adjacency matrix of the graph , where if nodes and are connected; otherwise, . In the semi-supervised setting, part of nodes are provided with labels , where denotes nodes with labels, and is the set of labels. Sensitive attributes of training nodes are required to achieve fairness of machine learning algorithms. In our setting, only a small set of nodes are provided with the sensitive attribute . The set of provided sensitive attributes is denoted by .

3.2. Datasets

Dataset Pokec-z Pokec-n NBA
# of nodes 67,797 66,569 403
# of node attributes 59 59 39
# of edges 882,765 729,129 16,570
Size of 500 500 100
Size of 200 200 50
Group ratio 1.84 2.46 2.77
# of inter-group edges 39,804 31,515 4,401
# of intra-group edges 842,961 697,614 12,169
Table 1. The statistics of datasets.

For the purpose of this study, we collect and sample datasets from Pokec and NBA. The details are described as below.

Pokec (Takac and Zabovsky, 2012): It is the most popular social network in Slovakia, which is very similar to Facebook and Twitter. This dataset contains anonymized data of the whole social network in 2012. User profiles of Pokec contain gender, age, hobbies, interest, education, working field and etc. The original Pokec dataset contains millions of users. Based on the provinces that users belong to, we sampled two datasets named as: Pokec-z and Pokec-n. Both Pokec-z and Pokec-n consist of users belonging to two major regions of the corresponding provinces. We treat the region as the sensitive attribute. The classification task is to predict the working field of the users.

NBA: This extended from a Kaggle dataset 111https://www.kaggle.com/noahgift/social-power-nba containing around 400 basketball players in NBA. The performance statistics of players in the 2016-2017 season and other various information e.g., nationality, age, and salary are provided. To obtain the graph that links the NBA players together, we collect the relationships of the NBA basketball players on Twitter with its official crawling API 222https://developer.twitter.com/en

. We binarize the nationality to two categories, i.e., U.S. players and oversea players, which is used as sensitive attribute. The classification task is to predict whether the salary of the player is over median.

For all the datasets, we eliminate nodes without any links with others. We randomly sample labels and sensitive attributes separately to get and . We randomly sample 25% and 50% of nodes containing both sensitive attributes and labels in Pokec-z, Pokec-n and NBA as validation sets and test sets. Note that the validation sets and test sets have no overlap with and . The key statistics of the datasets are given in Table 1

. Apart from the basic statistics, we also report the ratio of the majority and minority group and the number of edges linking the same group and different groups. It is evident from the table that: (i) skew exists in sensitive attributes; (ii) most of relationships are between users who share the same sensitive attribute.

3.3. Preliminaries of Graph Neural Networks

Graph neural networks (GNNs) utilize the node attributes and edges to learn a representation of the node . The goal of learning representation in node classification is to predict the node ’s label as . Current GNNs are neighborhood aggregation approaches, which will update the representations of the nodes with the representations of the neighborhood nodes. The representations after layers’ aggregation would capture the structural information of the -hop network neighborhoods. The updating process of the -th layer in GNN could be formulated as:



is the representation vector of the node

at -th layer and is a set of neighborhoods of .

3.4. Fairness Evaluation Metrics

In this subsection, we will present two definitions of fairness for the binary label and the sensitive attribute . denotes the prediction of the classifier : .

Definition 3.1 ().

(Statistical Parity (Dwork et al., 2012)). Statistical parity requires the predictions to be independent with the sensitive attribute , i.e., . It could be formally written as:

Definition 3.2 ().

(Equal Opportunity (Hardt et al., 2016)

). Equal opportunity requires the probability of an instance in a positive class being assigned to a positive outcome should be equal for both subgroup members. The property of equal opportunity is defined as:


The equal opportunity expects the classifier to give equal true positive rates across the subgroups. According to (Louizos et al., 2015; Beutel et al., 2017), we apply the following metrics to quantitatively evaluate statistical parity and equal opportunity:


where the probabilities are evaluated on the test set.

3.5. Discrimination in Graph Neural Networks

Dataset Metrics MLP MLP-e GCN GAT
Pokec-z ACC (%) 65.3 68.6 70.2 70.4
AUC (%) 71.3 74.8 77.2 76.7
(%) 3.8 6.9 9.9 9.1
(%) 2.2 4.0 9.1 8.4
Pokec-n ACC (%) 63.1 66.3 70.5 70.3
AUC (%) 68.2 72.4 75.1 75.1
(%) 3.3 8.7 9.6 9.4
(%) 7.1 9.9 12.8 12.0
NBA ACC (%) 63.6 66.1 71.2 71.9
AUC (%) 73.5 74.4 78.3 78.2
(%) 6.0 10.9 7.9 10.2
(%) 6.1 8.8 17.8 15.9
Table 2. Results of models w/ and w/o utilizing graph.

Various machine learning algorithms such as logistic regression

(Zafar et al., 2015), SVM (Zafar et al., 2015), and MLP (Edwards and Storkey, 2015) have been reported to have discrimination. The features of the instances may contain proxy variables of the sensitive attribute. It could result in biased predictions. For GNNs, edges in graph can bring linking bias, i.e., the misrepresentation due to the connections of users (Mehrabi et al., 2019). It has been proven that the embeddings of nodes within the connected component will be closer after one aggregation in GCN (Li et al., 2018; Wang and Leskovec, 2020). Since most of edges are intra-group as Table 1 shows, embeddings of nodes sharing the same sensitive attribute will be closer after -layer information aggregation. As a result, representations of the nodes may exhibit bias. Intuitively, similar discrimination also exists in other GNNs that aggregate information of neighborhoods.

To empirically demonstrate the existence of discrimination in GNNs, we make comparisons between the following models:

  • MLP: A multi-layer perception model trained on .

  • MLP-e: A MLP model utilizes graph structure by adding embeddings learned by deepwalk to the features.

  • GCN (Kipf and Welling, 2016): A state-of-the-art spectral graph neural network.

  • GAT (Veličković et al., 2017): A spatial graph neural network which utilizes attention to assign higher weights to more important edges.

For each model, we run the experiment 5 times. The classification results and discrimination scores on the test set are reported in Table 2. From the table, we observe that (i) the performance of GCN and GAT are much better than that of MLP, which is as expected because GCN and GAT adopt both node attributes and the graph structure for classification; (ii) Compared with MLP, models utilizing graph structure, i.e., GCN and GAT, perform significantly worse in terms of fairness, which verifies that bias exists in GNNs and the graph structure could further aggravate the discrimination.

3.6. Problem definition

Our preliminary analysis verifies that GNNs have severe bias issue. Thus, it is important to develop fair GNNs. Following existing work on fair models (Louizos et al., 2015; Feldman et al., 2015; Beutel et al., 2017; Xu et al., 2018), we focus on the binary class and binary sensitive attribute setting, i.e., both and can only be 0 or 1. We leave the extension to multi-class and multi-sensitive attribute setting as a future work. With the notations given in Section 3.1, the fair GNN problem is formally defined as:

Problem 1 ().

Given a graph , small labeled node set with the corresponding labels in , and a small set of nodes with corresponding sensitive attributes in , learn a fair GNN for fair node classification, i.e.,


where is the function we aim to learn and is the set of predicted labels for unlabeled nodes. should maintain high accuracy whilst satisfying the fairness criteria such as statistical parity.

4. methodology

In this section, we give the details of FairGNN. An illustration of the proposed framework is shown in Figure 1, which is composed of a GNN classifier , a GCN based sensitive attribute estimator and an adversary . The classifier takes as input for node classification. The sensitive attribute estimator is to predict the sensitive attributes for nodes whose sensitive attributes are unknown, which paves us a way to adopt adversarial learning to learn fair node representations and to regularize the predictions of . Specifically, the adversary aims to predict the known or estimated sensitive attributes by from the node representation learned by ; while aims to learn fair node representations that can fool the adversary to make wrong predictions. We theoretically prove that under mild conditions, such minmax game can guarantee that learned representations are fair. In addition to make the representations fair, we directly add a regularizer on the predictions of to guarantee that gives fair predictions. Next, we introduce each component in detail along with theoretical proof.

Figure 1. The overall framework of FairGNN.

4.1. The GNN Classifier

The GNN classifier takes as input and predicts node labels. The proposed framework FairGNN is flexible. Any GNNs that follow the structure of Eq.(1) can be used such as GCN (Kipf and Welling, 2016) and GAT (Veličković et al., 2017). Let denote the operation of aggregating and combining the information of node and its -hop neighborhoods through layers’ iterations in GNN classifier . For a GNN with layers, the representation of node of the final layer could be written as:


where represents the -hop neighborhoods of . To get the , i.e., the prediction of node , a linear classification layer is applied to as:


where is the weights of the linear classification layer and

is the sigmoid function. The loss function for training



where denotes the size of , represents the parameters of and is the groundtruth label of node .

4.2. Adversarial Debiasing with Estimator

The GNN classifier can make biased predictions because the learned representations of exhibit bias due to the node features, graph structure and aggregation mechanism of GNN. One way to make fair is to eliminate the bias of the final layer representations . Recently, adversarial debiasing has been proven to be effective in alleviating the bias of representations (Beutel et al., 2017; Edwards and Storkey, 2015; Liao et al., 2019; Madras et al., 2018). In the general process of adversarial debiasing, an adversary is used to predict sensitive attributes from the representations of the classifier; while the classifier is trained to learn representations to make the adversary unable to predict the sensitive attributes while keep high accuracy in the classification task. Such process requires abundant data samples with known sensitive attributes so that we can judge if the adversary can make accurate predictions or not.

However, in practice people are reluctant to share their sensitive attributes, which leads to small size . Lacking of data with labeled sensitive attributes would result in poor improvement in fairness even with adversarial debiasing. Though we have limited nodes with sensitive attributes, i.e., small , generally, nodes with similar sensitive attributes are more likely connected to each other, which makes it possible to accurately predict the sensitive attributes for nodes in using the graph and . Thus, we deploy a graph convolutional network to estimate the sensitive attribute of node whose sensitive attribute is unavailable. The large amount of estimated sensitive attributes would greatly benefit the adversarial debiasing. Note that it is important to use two separate GNNs for node label prediction and sensitive attribute prediction because we aim to learn fair representations for , i.e., does not contain the sensitive information. The objective function of training is


where is the predicted sensitive attribute of node by and is the set of parameters of .

With , we could get the estimation of the sensitive attributes of the nodes . We use to denote the set of sensitive attributes by combining and , i.e., . During the training process, for each node , the adversary tries to predict ’s sensitive attribute given the representation as ; while aims to learn node representation that makes the adversary unable to distinguish which sensitive group the node belong to. This min max game can be written as


where means sampling a node with sensitive attribute as 1 from . is the parameters of .

Theoretical Analysis. Since the size of is small, the estimation of sensitive attributes will introduce nonnegligible noise. The noise of the sensitive attributes may influence the adversarial debiasing. Thus, we conduct theoretical analysis to show that sensitive attributes containing noise could help to achieve statistical parity under mild conditions. Next, we give the details of the proof.

Proposition 4.1 ().

The global minimum of Eq.(11) is achieved if and only if , where and is final layer representation learned by the -layer GNN classifier .


According to Proposition 1. in (Goodfellow et al., 2014), the optimal adversary is . Then the min max game in Eq.(11) could be reformulated as minimizing this function:


The Jensen-Shannon divergence between two distributions is non-negative, and become zero if the two distributions are equal. Thus, only if , the objective function will reach the minimum, which completes our proof. ∎

Theorem 4.2 ().

Let denote the prediction of . Suppose:

  1. The estimated sensitive attribute and are independent conditioned on true sensitive attribute , i.e., ;

  2. .

If Eq.(11) reaches the global minimum, the GNN classifier will achieve statistical parity, i.e., .


Under the assumption that , we could obtain . From Proposition 4.1, we have when the algorithm converges, which is equivalent to . Together with , we arrive at


Reordering the terms in Eq.(13), we can get


Eq.(14) shows that at the global minimum under the assumption . Since , we could get . Thus, the statistical parity is achieved when Eq.(11) converges. ∎

In our proof, two assumptions are made. For the first assumption, since we use to predict the sensitive attributes and to get the latent representation , and and doesn’t share any parameters, it is generally true that is independent with the representation , i.e., . As for the second assumption, it will be satisfied when we have a reasonable estimator , i.e., doesn’t give random predictions.

4.3. Covariance Constraint

The instability of the training process of adversarial learning is well known (Arjovsky and Bottou, 2017). In adversarial debiasing, failure to coverage may result in a classifier with discrimination. To alleviate this issue, we add a covariance constraint (Zafar et al., 2015, 2017) on the output of to help the model achieve fairness. The covariance constraint has been explored in (Zafar et al., 2015, 2017) by minimizing the absolute covariance between users’ sensitive attributes and the signed distance from the users’ features to the decision boundary for fair linear classifiers . In our problem, only a small portion of users’ sensitive attributes are known and the decision boundary of GNN is hard to obtain. Thus, we propose to minimize the absolute covariance between the noisy sensitive attribute and prediction as


where indicates the absolute value.

Theoretical Analysis. Since is the absolute value of covariance between and , , i.e., the global minimum of , is the prerequisite that and are independent. Thus, we will show that is the prerequisite of the statistical parity under mild assumption with the following theorem.

Theorem 4.3 ().

Suppose that , when satisfies statistical parity, i.e. , is independent with and .


Through , we could get . Then, could be derived. When , the distribution would be:


Thus, is independent with when the statistical parity is achieved. Then, we can get . ∎

In the proof, we use the first assumption in Theorem 4.3, which is generally valid as discussed previously.

4.4. Final Objective Function of FairGNN

We now have for label prediction, for sensitive attribute estimation, with adversarial debiasing to force the node representations learned by are fair, and covariance constraint to further ensure that the prediction of is fair. Combining all these together, the final objective function could be formulated as:


where , , and are the parameters of classifier, estimator, and adversary, respectively. and are scalars to control the contributions of the covariance constraint and adversarial debiasing.

0:  , , , and .
0:  , , and
1:  Initialize by optimizing Eq.(10) w.r.t
2:  repeat
3:     Obtain the estimated sensitive attributes with
4:     Optimize the GNN classifier parameters , the adversary parameters , and the estimator parameters by Eq.(17).
5:  until convergence
6:  return  , , and
Algorithm 1 Training Algorithm of FairGNN.

4.5. An Training Algorithm of FairGNN

The training algorithm of FairGNN is presented in Algorithm 1. Specially, we first pretrain to ensure it meets the second assumption in Theorem 4.2. Sequentially, we optimize the whole model with Eq.(17) through the ADAM optimizer (Kingma and Ba, 2014). In the training process, we replace the hard labels in with soft labels, i.e., the probability produced by , to stabilize the adversarial learning (Salimans et al., 2016).

Dataset Metrics GCN GAT ALFR ALFR-e Debias Debias-e FCGE FairGCN FairGAT
Pokec-z ACC (%) 70.2 70.4 65.4 68.0 65.2 67.5 65.9 70.0 70.1
AUC (%) 77.2 76.7 71.3 74.0 71.4 74.2 71.0 76.7 76.5
(%) 9.9 9.1 2.8 5.8 1.9 4.7 3.1 0.9 0.5
(%) 9.1 8.4 1.1 2.8 1.9 3.0 1.7 1.7 0.8
Pokec-n ACC (%) 70.5 70.3 63.1 66.2 62.6 65.6 64.8 70.1 70.0
AUC (%) 75.1 75.1 67.7 71.9 67.9 71.7 69.5 74.9 74.9
(%) 9.6 9.4 3.05 4.1 2.4 3.6 4.1 0.8 0.6
(%) 12.8 12.0 3.9 4.6 2.6 4.4 5.5 1.1 0.8
NBA ACC (%) 71.2 71.9 64.3 66.0 63.1 65.6 66.0 71.1 71.5
AUC (%) 78.3 78.2 71.5 72.9 71.3 72.9 73.6 77.0 77.5
(%) 7.9 10.2 2.3 4.7 2.5 5.3 2.9 1.0 0.7
17.8 15.9 3.2 4.7 3.1 3.1 3.0 1.2 0.7
Table 3. The comparisons of our proposed methods with the baselines.

5. experiments

In this section, we conduct experiments to show the effectiveness of FairGNN for fair node classification. In particular, we aim to answer the following questions:

  • RQ1 Can the proposed FairGNN reduce the bias of GNNs while maintain high accuracy?

  • RQ2 How do the sensitive attribute estimator, adversarial loss, and covariance constraint affect FairGNN?

  • RQ3 Is FairGNN effective when different amount of sensitive attributes or labels are provided in the training set?

We use the same datasets introduced in Sec. 3.2 for all the experiments. Next, we will begin by introducing compared methods.

5.1. Compared Methods

We compare our proposed framework with GCN, GAT, and the following representative and state-of-the-art methods for fair classification and fair graph embedding learning:

  • [leftmargin=*]

  • ALFR (Edwards and Storkey, 2015)

    : This is a pre-processing method. A discriminator is applied to remove the sensitive information in the representations produced by a MLP-based autoencoder. Then, linear classifier is trained on the debiased representations.

  • ALFR-e: To utilize the graph structure information, ALFR-e concatenates the graph embeddings learned by deepwalk (Perozzi et al., 2014) with the user features in the ALFR.

  • Debias (Zhang et al., 2018): This is an in-processing fair classification method. It directly applies an discriminator on the estimated probability of classifier

    . It would make the probability distribution

    closer to .

  • Debias-e: Similar to the ALFR-e, we also add the deepwalk embeddings to the features used in Debias.

  • FCGE (Bose and Hamilton, 2019): FCGE is proposed to learn fair node embeddings in graph without node features through edge prediction. The sensitive information in the embeddings is filtered by discriminators.

ALFR and ALFR-e are trained with features of all the users , labels of , and the sensitive attributes of for fair classification. Debis and Debias-e require the sensitive attributes of labeled nodes, which is on contrary with our setting that could have no overlap with . Thus, we use the estimated labels of , features of , and labels of to train Debias and Debias-e. FCGE utilizes , labels of , and sensitive attributes of .

For FairGNN, we deploy a one hidden layer GCN for . The hidden dimension is set as 128. We use a linear classifier for . To verify that our framework is useful for various GNNs, we adopt both GCN and GAT as the backbone of the FairGNN classifier , which are named as FairGCN and FairGAT. In FairGCN, the GCN classifier contains one hidden layer with dimension 128. The GAT classifier in FairGAT also contains two layers in total. We set the number of heads as 1. The dimensions of the GAT classifiers’ hidden layer for Pokec-z, Pokec-n and NBA are 64, 64 and 32, respectively.

5.2. Fair Classification on Graph

To answer RQ1, we evaluate our proposed FairGNN in terms of fairness and classification performance. and are used to show the discrimination level, which are introduced in Section 3.4. The smaller and

are, the more fair the classifier is. Accuracy (ACC) and ROC AUC score are used to evaluate the classification performance. For all the models, we tune the hyperparameters on the training set via cross validation. For FairGCN, we set

to 100 and to 1. For FairGAT, is 2 and is 0.1. More details about hyperparameter selection will be discussed in Sec 5.5

. All the experiments are conducted 5 times. The mean and standard deviations for all the models on the three datasets are reported in Table

3. From the table, we make the following observations:

  • [leftmargin=*]

  • Compared with GCN and GAT, the general fair classification methods and graph embeddings learning method show poor performance in classification even with the help of graph information, while FairGCN and FairGAT perform very close to the based GNNs. This suggests the necessity of investigating fair classification algorithms on GNNs for accurate predictions;

  • Under the condition of limited sensitive information, baselines show obvious bias and the ones utilizing graph information are even worse. On the contrary, our proposed models obtain and that are close to 0, which indicates that the discrimination is basically eliminated; and

  • FairGAT is slightly better than FairGCN in Fairness. This is reasonable because the learnable edge coefficients in GAT could be helpful to reduce the weights of the edges that bring bias.

These observations demonstrate the effectiveness of our proposed framework in making fair and accurate predictions.

5.3. Ablation Study

To answer RQ2, we conduct ablation studies to understand the impacts of , adversarial loss, and covariance constraint.

5.3.1. Impact of

In our proposed framework, a GCN estimator is deployed to predict sensitive attributes for adversarial debiasing. To show the importance of the GCN estimator, we analyze it from two aspects. Firstly, to demonstrate the effectiveness of the noisy sensitive attributes, we eliminate the estimator and only use the provided sensitive attributes to get a variant denoted as FairGNNE. Secondly, to investigate how a weaker estimator would influence the fair classification, we train a variant by using MLP as the estimator. Hyperparameters of these variants are determined by cross validation with gird search. Specifically, we vary and among and , respectively. For each variant, the experiments are conducted 5 times. The average performance of fairness in terms of and node classificaiton in terms of AUC on Pockec-z are presented in Fig. 2(a) and (b), respectively. We only show the results on Pockec-z as we have similar observations on the other datasets. From the figures, we make the following observations:

  • [leftmargin=*]

  • The score of FairGNNE is much larger than that of FairGNN. which is because the provided sensitive attributes are inadequate. This shows that plays an important role in FairGNN; and

  • The performance of sensitive attribute prediction in terms of AUC for MLP estimator is 0.69, which is much lower than that of GCN estimator, which is 0.80. Though adopts a much weaker estimator than FairGNN, the performance in terms of fairness is slightly worse than FairGNN. This aligns with our theoretical analysis that doesn’t need to be very accurate. However, the marginal differences still indicate that too much noise in sensitive attributes may still slightly affect the fairness.

Figure 2. Comparisons between FairGNN and its variants.

5.3.2. Impacts of the adversarial debiasing and covariance constraint

To demonstrate the effects of the adversarial loss and covariance constraint, we train two variants of FairGNN, i.e., FairGNNA and FairGNNC, where FairGNNA means FairGNN without the adversarial loss, and FairGNNC means FiarGNN without covariance constraint. Similarly, for each variant, we run the experiment 5 times on Pokec-z and the average performances are shown in Figure 2. From the figure, we observe:

  • [leftmargin=*]

  • The scores for both FairGNNC and FairGNNA are much smaller than that of GNNs in Figure 2, which shows that covariance constraint and adversarial debiasing can improve fairness, respectively; and

  • The scores for both FairGNNC and FairGNNA are much larger than that of FairGNN, which implies that using both covariance constraint and adversarial debiasing can achieve better fairness. This is because they regularize the GNN from two different perspectives, i.e., adversarial debiasing regularizes on the node representations while covariance cosntraint is directly on the predictions for fair classification.

5.4. Impacts of Sizes of and

To answer RQ3, we study the impacts of the sizes of and on FairGAT. We set and based on cross validation. We vary as . Each experiment is conducted 5 times and the average results on Pokec-z with comparison to FairGATE and ALFR-e are shown in Fig. 3. From the figure, we observe that: (i) Generally, both FairGATE and ALFR-e have high discrimination scores when is small. They need plenty of data with sensitive attributes to become effective. FairGAT could get very low even when is as small as 200. This implies that FairGAT is insensitive to the size of data with sensitive attributes, which is because we have to estimate the sensitive attributes. Though extremely small would lead to a weak , we still have similar score as that when is large. This verifies our theoretical analysis that we can achieve good fairness with a reasonable ; (ii) FairGATE and ALFR-e decrease slightly in classification performance with the increasing of the size of , which is because more data with sensitive attribute would lead to a stricter regularization. In the contrary, FairGAT keeps high classification performance and even perform slightly better with more sensitive attributes. This is because the size of sensitive attributes used for training FairGAT are fixed to the size of , and less noise in the estimation of the sensitive attributes is helpful to better learn representations for classification.

Similarly, we vary as and each experiment is run for 5 times. The average results on Pokec-z are reported in Figure 3(a). We only report the results on Pokec-z as we have simialr observations on other datasets. From the figure, we observe that: FairGAT consistently shows effectiveness in eliminating discrimination. The drop in classification performance is marginal. This demonstrates that our proposed method could achieve fairness while keep high accuracy in general scenarios which correspond to various sizes of and .

Figure 3. Impacts of the size of to FairGAT.
Figure 4. Impacts of the size of to FairGAT.

5.5. Parameter Sensitivity

There are two important hyperparameters in our proposed model, .i.e., controlling the influence of the adversary to the GNN classifier, while controlling the contribution of the covariance constraint to ensure fairness. To investigate the parameter sensitivity and find the ranges that achieve high accuracy with low discrimination score, we train FairGAT models on Pokec-z with various hyperparameters. More specifically, we alter the values of and among and . The results are presented in Figure 5. From Figure 5 (fig:abl_roc), we can find that when and the classification performance is almostly unaffected. Once and is too large, the classifier’s performance will decay rapidly. The impacts of the hyperparameters to the discrimination score are presented in Figure 5 (fig:abl_sp). When we increase the value of , will firstly decrease as expected. Then, it would increase when the value of is too large. Because it would be difficult to optimize the GNN classifier to the global minimum when the contribution of the adversary is extremely high. As for , the discrimination score would consistently reduce when we increase its value. Combining the two figures, we could determine that when and , the GNN classifier achieves fairness and maintains high node classification accuracy.

(a) (%)
(b) ROC AUC (%)
Figure 5. Parameter sensitivity analysis.

6. conclusion and future work

In this paper, we study a novel problem of fair GNN learning with limited sensitive information. We empirically demonstrate that GNNs exhibit severe bias. We propose a novel and flexible framework FairGNN which is able to significantly alleviate the bias issue of GNNs meanwhile maintain high performance on node classification. FairGNN adopts a sensitive attribute estimator to alleviate the issue of lacking sensitive attribute information. With the estimated sensitive attributes, FairGNN designs adversarial debiasing and covariance constraint to regularize the GNN to have fair node representations and predictions, respectively. We theoretically show that FairGNN can reduce the bias. Experiment results on real-world datasets demonstrate the effectiveness of the proposed framework in terms of both fairness and classification performance. There are several interesting directions which need further investigation. First, we assume the provided sensitive attributes are clean. However, for some applications in social media, users might randomly input sensitive attributes such as gender due to privacy concern. Thus, we will extend FairGNN to deal with limited and inaccurate sensitive information. Second, the experiments show that the edges are possible to bring bias. Thus, we will also explore methods which add/delete links in graphs to improve the fairness and classification performance of FairGNN.


  • M. Arjovsky and L. Bottou (2017)

    Towards principled methods for training generative adversarial networks

    arXiv preprint arXiv:1701.04862. Cited by: §4.3.
  • R. v. d. Berg, T. N. Kipf, and M. Welling (2017) Graph convolutional matrix completion. arXiv preprint arXiv:1706.02263. Cited by: §1, §2.1.
  • A. Beutel, J. Chen, Z. Zhao, and E. H. Chi (2017) Data decisions and theoretical implications when adversarially learning fair representations. arXiv preprint arXiv:1707.00075. Cited by: §1, §1, §1, §2.2, §3.6, Definition 3.2, §4.2.
  • A. J. Bose and W. L. Hamilton (2019) Compositional fairness constraints for graph embeddings. arXiv preprint arXiv:1905.10674. Cited by: §1, §2.2, 5th item.
  • J. Bruna, W. Zaremba, A. Szlam, and Y. LeCun (2013) Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6203. Cited by: §1, §2.1.
  • J. Buolamwini and T. Gebru (2018) Gender shades: intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency, pp. 77–91. Cited by: §1.
  • J. Chen, T. Ma, and C. Xiao (2018) Fastgcn: fast learning with graph convolutional networks via importance sampling. arXiv preprint arXiv:1801.10247. Cited by: §2.1.
  • W. Chiang, X. Liu, S. Si, Y. Li, S. Bengio, and C. Hsieh (2019) Cluster-gcn: an efficient algorithm for training deep and large graph convolutional networks. In SIGKDD, pp. 257–266. Cited by: §2.1.
  • E. Creager, D. Madras, J. Jacobsen, M. A. Weis, K. Swersky, T. Pitassi, and R. Zemel (2019) Flexibly fair representation learning by disentanglement. arXiv preprint arXiv:1906.02589. Cited by: §1, §1, §2.2.
  • M. Defferrard, X. Bresson, and P. Vandergheynst (2016) Convolutional neural networks on graphs with fast localized spectral filtering. In NeurIPS, pp. 3844–3852. Cited by: §2.1.
  • Y. Dong, O. Lizardo, and N. V. Chawla (2016) Do the young live in a” smaller world” than the old? age-specific degrees of separation in a large-scale mobile communication network. arXiv preprint arXiv:1606.07556. Cited by: §1.
  • C. Dwork, M. Hardt, T. Pitassi, O. Reingold, and R. Zemel (2012) Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, pp. 214–226. Cited by: §1, §2.2, Definition 3.1.
  • H. Edwards and A. Storkey (2015) Censoring representations with an adversary. arXiv preprint arXiv:1511.05897. Cited by: §1, §1, §2.2, §3.5, §4.2, 1st item.
  • M. Feldman, S. A. Friedler, J. Moeller, C. Scheidegger, and S. Venkatasubramanian (2015) Certifying and removing disparate impact. In proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pp. 259–268. Cited by: §2.2, §3.6.
  • J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl (2017) Neural message passing for quantum chemistry. arXiv preprint arXiv:1704.01212. Cited by: §2.1.
  • I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio (2014) Generative adversarial nets. In NeurIPS, pp. 2672–2680. Cited by: §4.2.
  • A. Grover and J. Leskovec (2016) Node2vec: scalable feature learning for networks. In SIGKDD, pp. 855–864. Cited by: §2.2.
  • T. Hamaguchi, H. Oiwa, M. Shimbo, and Y. Matsumoto (2017) Knowledge transfer for out-of-knowledge-base entities: a graph neural network approach. arXiv preprint arXiv:1706.05674. Cited by: §1, §2.1.
  • W. Hamilton, Z. Ying, and J. Leskovec (2017) Inductive representation learning on large graphs. In NeurIPS, pp. 1024–1034. Cited by: §1, §2.1.
  • M. Hardt, E. Price, and N. Srebro (2016)

    Equality of opportunity in supervised learning

    In NeurIPS, pp. 3315–3323. Cited by: §2.2, Definition 3.2.
  • M. Henaff, J. Bruna, and Y. LeCun (2015) Deep convolutional networks on graph-structured data. arXiv preprint arXiv:1506.05163. Cited by: §2.1.
  • F. Kamiran and T. Calders (2009) Classifying without discriminating. In 2009 2nd International Conference on Computer, Control and Communication, pp. 1–6. Cited by: §1, §2.2.
  • F. Kamiran and T. Calders (2012) Data preprocessing techniques for classification without discrimination. Knowledge and Information Systems 33 (1), pp. 1–33. Cited by: §1, §2.2.
  • T. Kamishima, S. Akaho, and J. Sakuma (2011) Fairness-aware learning through regularization approach. In 2011 IEEE 11th International Conference on Data Mining Workshops, pp. 643–650. Cited by: §2.2.
  • D. P. Kingma and J. Ba (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §4.5.
  • T. N. Kipf and M. Welling (2016) Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907. Cited by: §1, §2.1, 3rd item, §4.1.
  • R. Levie, F. Monti, X. Bresson, and M. M. Bronstein (2018) Cayleynets: graph convolutional neural networks with complex rational spectral filters. IEEE Transactions on Signal Processing 67 (1), pp. 97–109. Cited by: §2.1.
  • Q. Li, Z. Han, and X. Wu (2018)

    Deeper insights into graph convolutional networks for semi-supervised learning

    In AAAI, Cited by: §3.5.
  • J. Liao, C. Huang, P. Kairouz, and L. Sankar (2019) Learning generative adversarial representations (gap) under fairness and censoring constraints. arXiv preprint arXiv:1910.00411. Cited by: §1, §2.2, §4.2.
  • F. Locatello, G. Abbati, T. Rainforth, S. Bauer, B. Schölkopf, and O. Bachem (2019) On the fairness of disentangled representations. In NeurIPS, pp. 14584–14597. Cited by: §1, §2.2.
  • C. Louizos, K. Swersky, Y. Li, M. Welling, and R. Zemel (2015) The variational fair autoencoder. arXiv preprint arXiv:1511.00830. Cited by: §1, §2.2, §3.6, Definition 3.2.
  • M. Madden, A. Lenhart, S. Cortesi, U. Gasser, M. Duggan, A. Smith, and M. Beaton (2013) Teens, social media, and privacy. Pew Research Center 21 (1055), pp. 2–86. Cited by: §1.
  • D. Madras, E. Creager, T. Pitassi, and R. Zemel (2018) Learning adversarially fair and transferable representations. arXiv preprint arXiv:1802.06309. Cited by: §1, §1, §2.2, §4.2.
  • N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, and A. Galstyan (2019) A survey on bias and fairness in machine learning. arXiv preprint arXiv:1908.09635. Cited by: §1, §3.5.
  • M. Niepert, M. Ahmed, and K. Kutzkov (2016) Learning convolutional neural networks for graphs. In ICML, pp. 2014–2023. Cited by: §2.1.
  • A. Olteanu, C. Castillo, F. Diaz, and E. Kiciman (2019) Social data: biases, methodological pitfalls, and ethical boundaries. Frontiers in Big Data 2, pp. 13. Cited by: §1.
  • B. Perozzi, R. Al-Rfou, and S. Skiena (2014) Deepwalk: online learning of social representations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 701–710. Cited by: 2nd item.
  • G. Pleiss, M. Raghavan, F. Wu, J. Kleinberg, and K. Q. Weinberger (2017) On fairness and calibration. In NeurIPS, pp. 5680–5689. Cited by: §2.2.
  • T. A. Rahman, B. Surma, M. Backes, and Y. Zhang (2019) Fairwalk: towards fair graph embedding.. In IJCAI, pp. 3289–3295. Cited by: §1, §1, §2.2.
  • T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen (2016) Improved techniques for training gans. In NeurIPS, pp. 2234–2242. Cited by: §4.5.
  • P. Sattigeri, S. C. Hoffman, V. Chenthamarakshan, and K. R. Varshney (2019) Fairness gan: generating datasets with fairness properties using a generative adversarial network. IBM Journal of Research and Development 63 (4/5), pp. 3–1. Cited by: §2.2.
  • J. Song, P. Kalluri, A. Grover, S. Zhao, and S. Ermon (2018) Learning controllable fair representations. arXiv preprint arXiv:1812.04218. Cited by: §1, §2.2.
  • H. Suresh and J. V. Guttag (2019) A framework for understanding unintended consequences of machine learning. arXiv preprint arXiv:1901.10002. Cited by: §1.
  • L. Takac and M. Zabovsky (2012) Data analysis in public social networks. In International scientific conference and international workshop present day trends of innovations, Vol. 1. Cited by: §3.2.
  • P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio (2017) Graph attention networks. arXiv preprint arXiv:1710.10903. Cited by: §1, §2.1, 4th item, §4.1.
  • H. Wang and J. Leskovec (2020) Unifying graph convolutional neural networks and label propagation. arXiv preprint arXiv:2002.06755. Cited by: §3.5.
  • Z. Wang, Q. Lv, X. Lan, and Y. Zhang (2018) Cross-lingual knowledge graph alignment via graph convolutional networks. In EMNLP, pp. 349–357. Cited by: §1, §2.1.
  • D. Xu, S. Yuan, L. Zhang, and X. Wu (2018) Fairgan: fairness-aware generative adversarial networks. In 2018 IEEE International Conference on Big Data (Big Data), pp. 570–575. Cited by: §2.2, §3.6.
  • D. Xu, S. Yuan, L. Zhang, and X. Wu (2019) FairGAN+: achieving fair data generation and classification through generative adversarial nets. In 2019 IEEE International Conference on Big Data (Big Data), pp. 1401–1406. Cited by: §2.2.
  • L. Yao, C. Mao, and Y. Luo (2019) Graph convolutional networks for text classification. In AAAI, Vol. 33, pp. 7370–7377. Cited by: §1, §2.1.
  • R. Ying, R. He, K. Chen, P. Eksombatchai, W. L. Hamilton, and J. Leskovec (2018) Graph convolutional neural networks for web-scale recommender systems. In SIGKDD, pp. 974–983. Cited by: §1, §2.1.
  • M. B. Zafar, I. Valera, M. Gomez Rodriguez, and K. P. Gummadi (2017) Fairness beyond disparate treatment & disparate impact: learning classification without disparate mistreatment. In WWW, pp. 1171–1180. Cited by: §1, §4.3.
  • M. B. Zafar, I. Valera, M. G. Rodriguez, and K. P. Gummadi (2015) Fairness constraints: mechanisms for fair classification. arXiv preprint arXiv:1507.05259. Cited by: §1, §2.2, §3.5, §4.3.
  • R. Zemel, Y. Wu, K. Swersky, T. Pitassi, and C. Dwork (2013) Learning fair representations. In ICML, pp. 325–333. Cited by: §1, §2.2.
  • B. H. Zhang, B. Lemoine, and M. Mitchell (2018) Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp. 335–340. Cited by: §1, §1, §2.2, 3rd item.
  • L. Zhang, Y. Wu, and X. Wu (2017) Achieving non-discrimination in data release. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1335–1344. Cited by: §1, §2.2.