Blind Justice: Fairness with Encrypted Sensitive Attributes

by   Niki Kilbertus, et al.

Recent work has explored how to train machine learning models which do not discriminate against any subgroup of the population as determined by sensitive attributes such as gender or race. To avoid disparate treatment, sensitive attributes should not be considered. On the other hand, in order to avoid disparate impact, sensitive attributes must be examined, e.g., in order to learn a fair model, or to check if a given model is fair. We introduce methods from secure multi-party computation which allow us to avoid both. By encrypting sensitive attributes, we show how an outcome-based fair model may be learned, checked, or have its outputs verified and held to account, without users revealing their sensitive attributes.



page 4


Learning Fair Models without Sensitive Attributes: A Generative Approach

Most existing fair classifiers rely on sensitive attributes to achieve f...

Causal Multi-Level Fairness

Algorithmic systems are known to impact marginalized groups severely, an...

Fair Outlier Detection

An outlier detection method may be considered fair over specified sensit...

Multiaccurate Proxies for Downstream Fairness

We study the problem of training a model that must obey demographic fair...

Fair Learning with Private Demographic Data

Sensitive attributes such as race are rarely available to learners in re...

CryptoCredit: Securely Training Fair Models

When developing models for regulated decision making, sensitive features...

Fairness in Machine Learning

Machine learning based systems are reaching society at large and in many...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Concerns are rising that machine learning systems which make or support important decisions affecting individuals—such as car insurance pricing, résumé filtering or recidivism prediction—might illegally or unfairly discriminate against certain subgroups of the population (Schreurs et al., 2008; Calders & Žliobaitė, 2012; Barocas & Selbst, 2016). The growing field of fair learning seeks to formalize relevant requirements, and through altering parts of the algorithmic decision-making pipeline, to detect and mitigate potential discrimination (Friedler et al., 2016).

Most legally-problematic discrimination centers on differences based on sensitive attributes, such as gender or race (Barocas & Selbst, 2016). The first type, disparate treatment (or direct discrimination), occurs if individuals are treated differently according to their sensitive attributes (with all others equal). To avoid disparate treatment, one should not inquire about individuals’ sensitive attributes. While this has some intuitive appeal and justification (Grgić-Hlača et al., 2018), a significant concern is that sensitive attributes may often be accurately predicted (“reconstructed”) from non-sensitive features (Dwork et al., 2012). This motivates measures to deal with the second type of discrimination.

Disparate impact (or indirect discrimination) occurs when the outcomes of decisions disproportionately benefit or hurt individuals from subgroups with particular sensitive attribute settings without appropriate justification. For example, firms deploying car insurance telematics devices (Handel et al., 2014) build up high dimensional pictures of driving behavior which might easily proxy for sensitive attributes even when they are omitted. Much recent work in fair learning has focused on approaches to avoiding various notions of disparate impact (Feldman et al., 2015; Hardt et al., 2016; Zafar et al., 2017c).

In order to check and enforce such requirements, the modeler must have access to the sensitive attributes for individuals in the training data—however, this may be undesirable for several reasons (Žliobaitė & Custers, 2016). First, individuals are unlikely to want to entrust sensitive attributes to modelers in all application domains. Where applications have clear discriminatory potential, it is understandable that individuals may be wary of providing sensitive attributes to modelers who might exploit them to negative effect, especially with no guarantee that a fair model will indeed be learned and deployed. Even if certain modelers themselves were trusted, the wide provision of sensitive data creates heightened privacy risks in the event of a data breach.

Further, legal barriers may limit collection and processing of sensitive personal data. A timely example is the EU’s General Data Protection Regulation (GDPR), which contains heightened prerequisites for the collection and processing of some sensitive attributes. Unlike other data, modelers cannot justify using sensitive characteristics in fair learning with their “legitimate interests”—and instead will often need explicit, freely given consent (Veale & Edwards, 2018).

One way to address these concerns was recently proposed by Veale & Binns (2017). The idea is to involve a highly trusted third party, and may work well in some cases. However, there are significant potential difficulties: individuals must disclose their sensitive attributes to the third party (even if an individual trusts the party, she may have concerns that the data may somehow be obtained or hacked by others, e.g., Graham, 2017); and the modeler must disclose their model to the third party, which may be incompatible with their intellectual property or other business concerns.


We propose an approach to detect and mitigate disparate impact without disclosing readable access to sensitive attributes. This reflects the notion that decisions should be blind to an individual’s status—depicted in courtrooms by a blindfolded Lady Justice holding balanced scales (Bennett Capers, 2012). We assume the existence of a regulator with fairness aims (such as a data protection authority or anti-discrimination agency). With recent methods from secure multi-party computation (MPC), we enable auditable fair learning while ensuring that both individuals’ sensitive attributes and the modeler’s model remain private to all other parties—including the regulator. Desirable fairness and accountability applications we enable include:

  1. [leftmargin=1em,topsep=0pt,itemsep=0ex,partopsep=1ex,parsep=1ex]

  2. Fairness certification. Given a model and a dataset of individuals, check that the model satisfies a given fairness constraint (we consider several notions from the literature, see Section 2.2); if yes, generate a certificate.

  3. Fair model training. Given a dataset of individuals, learn a model guaranteed and certified to be fair.

  4. Decision verification. A malicious modeler might go through fair model training, but then use a different model in practice. To address such accountability concerns (Kroll et al., 2016), we efficiently provide for an individual to challenge a received outcome, verifying that it matches the outcome from the previously certified model.

We rely on recent theoretical developments in MPC (see Section 3) which we extend to admit linear constraints in order to enforce fairness requirements. These extensions may be of independent interest. We demonstrate the real-world efficacy of our methods, and shall make our code publicly available.

2 Fairness and Privacy Requirements

Here we formalize our setup and requirements.

2.1 Assumptions and Incentives

We assume three categories of participants: a modeler M, a regulator Reg, and users

. For each user, we consider a vector of sensitive features (or attributes, we use the terms interchangeably)

(e.g., ethnicity or gender) which might be a source of discrimination, and a vector of non-sensitive features (discrete or real). Additionally, each user has a non-sensitive feature which the modeler M would like to predict—the label (e.g., loan default). In line with current work in fair learning, we assume that all and attributes are binary, though our MPC approach could be extended to multi-label settings. The source of societal concern is that sensitive attributes  are potentially correlated with  or .

Modeler M wishes to train a model , which accurately maps features  to labels , in a supervised fashion. We assume M needs to keep the model private for intellectual property or other business reasons. The model does not use sensitive information as input to prevent disparate treatment (direct discrimination).

For each user , M observes or is provided . The sensitive information in  is required to ensure  meets a given disparate impact fairness condition  (see Section 2.2). While each user  wants  to meet , they also wish to keep  private from all other parties. The regulator Reg aims to ensure that M deploys only models that meet fairness condition . It has no incentive to collude with M (if collusion were a concern, more sophisticated cryptographic protocols would be required). Further, the modeler M might be legally obliged to demonstrate to the regulator Reg that their model meets fairness condition  before it can be publicly deployed. As part of this, Reg also has a positive duty to enable the training of fair models.

In Section 2.3, we define and address three fundamental problems in our setup: certification, training, and verification. For each problem, we present its functional goal and its privacy requirements. We refer to and as the non-sensitive and sensitive data, respectively. In Section 2.2, we first provide necessary background on various notions of fairness that have been explored in the fair learning literature.

2.2 Fairness Criteria

In large part, works that formalize fairness in machine learning do so by balancing a certain condition between groups of people with different sensitive attributes, versus . Several possible conditions have been proposed. Popular choices include (where and is the prediction of a machine learning model):


Respectively, these consider equality of: (1) accuracy, (2) true positive rate, (3) true negative rate, (4) positive predicted value, (5) negative predicted value, or (6) acceptance rate. Works which use these or related notions include (Hardt et al., 2016; Zafar et al., 2017c, a, b).

In this work we focus on a variant of eq. (6), formulated as a constrained optimization problem by Zafar et al. (2017c) mimicking the -rule (Biddle, 2006): for any binary protected attribute , it aims to achieve


We believe that in future work, a similar MPC approach could also be used for conditions (1), (2) or (3)—i.e., all the other measures which, to our knowledge, have been addressed with efficient standard (non-private) methods.

Figure 1: Our setup for Fairness certification (Left), Fair model training (Center), and Decision verification (Right).

2.3 Certification, Training, and Verification

Fairness certification.

Given a notion of fairness , the modeler M would like to work with the regulator Reg to obtain a certificate that model is fair. To do so, we propose that users send their non-sensitive data to Reg; and send encrypted versions of their sensitive data to both M and Reg. Neither M nor Reg can read the sensitive data. However, we can design a secure protocol between M and Reg (described in Section 3) to certify if the model is fair. This setup is shown in Figure 1 (Left).

While both Reg and M learn the outcome of the certification, we require the following privacy constraints: (C1) privacy of sensitive user data: no one other than ever learns in the clear, (C2) model secrecy: only M learns in the clear, and (C3) minimal disclosure of to Reg: only Reg learns in the clear.

Fair model training.

How can a modeler M learn a fair model without access to users’ sensitive data ? We propose to solve this by having users send their non-sensitive data to M and to distribute encryptions of their sensitive data to M and Reg as in certification. We shall describe a secure MPC protocol between M and Reg to train a fair model privately. This setup is shown in Figure 1 (Center).

Privacy constraints: (C1) privacy of sensitive user data, (C2) model secrecy, and (C3) minimal disclosure of to M.

Decision verification.

Assume that a malicious M has had model  successfully certified by Reg as above. It then swaps for another potentially unfair model in the real world. When a user receives a decision , e.g., her mortgage is denied, she can then challenge that decision by asking Reg for a verification . The verification involves M and Reg, and consists of verifying that , where is the user’s non-sensitive data. This ensures that the user would have been subject to the same result with the certified model , even if and is not fair. Hence, while there is no simple technical way to prevent a malicious M from deploying an unfair model, it will get caught if a user challenges a decision that would differ under . This setup is shown in Figure 1 (Right).

Privacy constraint: While Reg and the user learn the outcome of the verification, we require (C1) privacy of sensitive user data, and (C2) model secrecy.

2.4 Design Choices

We use a regulator for several reasons. Given fair learning is of most benefit to vulnerable individuals, we do not wish to deter adoption with high individual burdens. While MPC could be carried out without the involvement of a regulator, using all users as parties, this comes at a significantly greater computational cost. With current methods, taking that approach would be unrealistic given the size of the user-base in many domains of concern, and would furthermore require all users to be online simultaneously. Introducing a regulator removes these barriers and leaves users’ computational burden at a minimum level, with envisaged applications practical with only their web browsers.

In cases where users are uncomfortable sharing with either Reg or M, it is trivial to extend all three tasks such that all of remain private throughout, with the computation cost increasing only by a factor of 2. This extension would sometimes be desirable as it restricts the view of M to the final model, prohibiting inferences about when is known. However, this setup hinders exploratory data analysis by the modeler which might promote robust model-building, and, in the case of verification, validation by the regulator that user-provided data is correct.

3 Our Solution

Our proposed solution to these three problems is to use Multi-Party Computation (MPC). Before we describe how it can be applied to fair learning, we first present the basic principles of MPC, as well as its limitations particularly in the context of machine learning applications.

3.1 MPC for Machine Learning

Multi-Party Computation protocols allow two parties  and  holding secret values  and  to evaluate an agreed-upon function , via in a way in which the parties (either both or one of them) learn only . For example, if , then the parties would learn which of their values is bigger, but nothing else.111The function is if its argument is true and otherwise. This corresponds to the well-known Yao’s millionaires problem: two millionaires want to conclude who is richer without disclosing their wealth to each other. The problem was introduced by Andrew Yao in 1982, and kicked off the area of multi-party computation in cryptography.

In our setting—instead of a simple comparison as in the millionaires problem— will be either (i) a procedure to check the fairness of a model and certify it, (ii) a machine learning training procedure with fairness constraints, or (iii) a model evaluation to verify a decision. The two parties involved in our computation are the modeler M and the regulator Reg. The inputs depend on the case (see Figure 1).

As generic solutions do not yet scale to real-world data analysis tasks, one typically has to tailor custom protocols to the desired functionality. This approach has been followed successfully for a variety of machine learning tasks such as logistic and linear regression

(Nikolaenko et al., 2013b; Gascón et al., 2017; Mohassel & Zhang, 2017)

, neural network training

(Mohassel & Zhang, 2017) and evaluation (Juvekar et al., 2018; Liu et al., 2017), matrix factorization (Nikolaenko et al., 2013a)

, and principal component analysis

(Al-Rubaie et al., 2017). In the next section we review challenges beyond scalability issues that arise when implementing machine learning algorithms in MPC.

3.2 Challenges in Multi-Party Machine Learning

MPC protocols can be classified into two groups depending on whether the target function is represented as either a Boolean or arithmetic circuit. All protocols proceed by having the parties jointly evaluate the circuit, processing it gate by gate while keeping intermediate values hidden from both parties by means of a secret sharing scheme. While representing functions as circuits can be done without losing expressiveness, it means certain operations are impractical. In particular, algorithms that execute different branches depending on the input data will explode in size when implemented as circuits, and in some cases lose their run time guarantees (e.g., consider binary search).

Crucially, this applies to floating-point arithmetic

. While this is work in progress, state-of-the-art MPC floating-point arithmetic implementations take more than 

milliseconds to multiply two bit numbers (Demmler et al., 2015a, Table 4), which is prohibitive for our applications. Hence, machine learning MPC protocols are limited to fixed-point arithmetic. Overcoming this limitation is a key challenge for the field. Another necessity for the feasibility of MPC is to approximate non-linear functions such as the sigmoid, ideally by (piecewise) linear functions.

3.3 Our MPC Protocols

Input sharing.

To implement the functionality from Figure 1, we first need a secure procedure for the users to secret share a sensitive value, for example her race, with the modeler M and the regulator Reg. We use additive secret sharing. A value  is represented in a finite domain —we use . To share , the user samples a value  from  uniformly at random, and sends to M and  to Reg. While can be reconstructed (and subsequently operated on) inside the MPC computation by means of a simple addition, each share on its own does not reveal anything  (other than that it is in 

). One can think of arithmetic sharing as a “distributed one-time pad”.

In Figure 1, we now reinterpret the key held by Reg and the encrypted by M, as their corresponding shares of the sensitive attributes and denote them by and respectively. The idea of privately outsourcing computation to two non-colluding parties in this way is recurrent in MPC, and often referred to as the two-server model (Mohassel & Zhang, 2017; Gascón et al., 2017; Nikolaenko et al., 2013b; Al-Rubaie et al., 2017).

Signing and checking a model.

Note that certification and verification partly correspond to sub-procedures of the fair training task: during training we check the fairness constraint , and repeatedly evaluate partial models on the training dataset (using gradient descent). Hence, certification and verification do not add technical difficulties over training, which is described in detail in Section 4. However, for verification, we still need to “sign” the model, i.e., Reg should obtain a signature as a result of model certification, see Figure 1 (Left). This signature is used to check in the verification phase, whether a given model from M satisfies for a certified fair model (in which case

with high probability). Moreover, we need to preserve the secrecy of the model, i.e.,

Reg should not be able to recover from . These properties, given that the space of models is large, calls for a cryptographic hash function, such as SHA-256.

Additionally, in our functionality, the hash of should be computed inside MPC, to hide from Reg. Fortunately, cryptographic hashes such as SHA-256 are a common benchmark functionality in MPC, and their execution is highly optimized. More concretely, the overhead of computing , which needs to be done both for certification and verification is of the order of fractions of a second (Keller et al., 2013, Figure 14). While cryptographic hash functions have various applications in MPC, we believe the application to machine learning model certification is novel.

Hence, certification is implemented in MPC as a check that satisfies the criterion , followed by the computation of . On the other hand, for verification, the MPC protocol first computes the signature of the model provided by M, and then proceeds with a prediction as long as the computed signature matches the one obtained by Reg in the verification phase. An alternative solution is possible based on symmetric encryption under a shared key, as highly efficient MPC implementations of block ciphers such as AES are available (Keller et al., 2017).

Fair training.

To realize the fair training functionality from the previous section, we follow closely the techniques recently introduced by Mohassel & Zhang (2017)

. Specifically, we extend their custom MPC protocol for logistic regression to additionally handle linear constraints. This extension may be of independent interest, and has applications for privacy-preserving machine learning beyond fairness. The concrete technical difficulties in achieving this goal, and how to overcome them, are presented in the next section. The formal privacy guarantees of our fair training protocol are stated in the following proposition.

Proposition 1.

For non-colluding M and Reg, our protocol implements the fair model training functionality satisfying constraints (C1)-(C3) in Section 2.3 in the presence of a semi-honest adversary.

The proof holds in the random oracle model, as a standard simulation argument combining several MPC primitives (Mohassel & Zhang, 2017; Gascón et al., 2017). It leverages security of arithmetic sharing, garbled circuits, and oblivious transfer protocols in the semi-honest model (Goldreich et al., 1987). A general introduction to MPC, as well as a description of the relevant techniques from (Mohassel & Zhang, 2017) used in our protocol, can be found in Section A in the appendix.

4 Technical Challenges of Fair Training

We now present our tailored approaches for learning and evaluating fair models with encrypted sensitive attributes. We do this via the following contributions:

  • [leftmargin=*, noitemsep, topsep=0pt]

  • We argue that current optimization techniques for fair learning algorithms are unstable for fixed-point data, which is required by our MPC techniques.

  • We describe optimization schemes that are well-suited for learning over fixed-point number representations.

  • We combine tricks to approximate non-linear functions with specialized operations to make fixed-point arithmetic feasible and avoid over- and under-flows.

The optimization problem at hand is to learn a classifier subject to a (often convex) fairness constraint :


where is a loss term (the logistic loss in this work). We collect user data from  into matrices  and a label vector .

Zafar et al. (2017c) use a convex approximation of the -rule, see eq. (7), for linear classifiers to derive the constraint:


where  is the matrix of all and  is a constant vector corresponding to the tightness of the fairness constraint. Here, is the mean of all inputs . With , the constraint reads , where the absolute value is taken element-wise.

4.1 Current Techniques

To solve the optimization problem in eq. (8), with the fairness function  in eq. (9), Zafar et al. (2017c) use Sequential Least Squares Programming (SLSQP). This technique works by reformulating eq. (8) as a sequence of Quadratic Programs (QPs). After solving each QP, their algorithm uses the Han-Powell method, a quasi-Newton method that iteratively approximates the Hessian of the objective function via the update

where and is the Lagrangian of eq. (8). Finally, .

There are two issues with this approach from an MPC perspective. First, solving a sequence of QPs is prohibitively time-consuming in MPC. Second, while the above Han-Powell update performs well on floating-point data, the two divisions by non-constant, non-integer numbers easily underflow or overflow with fixed-point numbers.

4.2 Fixed-Point-Friendly Optimization Techniques

Instead, to solve the optimization problem in eq. (8

), we perform stochastic gradient descent and experiment with the following techniques to incorporate the constraints.

Lagrangian multipliers. Here we minimize

using stochastic gradient descent, i.e., alternating updates and , where  are the learning rates.

Projected gradient descent. For this method, consider specifically the -rule based notion . We first define  as the matrix consisting of the rows of  for which , i.e., where the constraint is active. In each step, we project the computed gradient of the binary-cross-entropy loss —either of a single example or averaged over a minibatch—back into the constraint set, i.e.,


Interior point log barrier (Boyd & Vandenberghe, 2004). We can approximate eq. (8) for the -rule constraint  by: minimize , where  is the th row of . The parameter  trades off the approximation of the true objective ( for  and for ) and the smoothness of the objective function. Throughout training  is increased, allowing the solution to move closer to the boundary. As the gradient of the objective has a simple closed form representation, we can perform regular (stochastic) gradient descent.

After extensive experiments (see Section 5

) we found the Lagrangian multipliers technique to work best, both in yielding high accuracies, reliably staying within the constraints and being robust to hyperparameter changes such as learning rates or the batch size. For a proof of concept, in Section 

5 we focus on the -rule, i.e., eq. (9). Note that the gradients for eq. (2) and eq. (3

) take a similarly simple form, i.e., balancing the true positive or true negative rates (corresponding to equal opportunity or equal odds) is simple to implement for the Lagrangian multiplier technique, but harder for projected gradient descent. However, these fairness notions are more expensive as we have to compute 

for each update step, instead of pre-computing it once at the beginning of training, see Algorithm 1 in the appendix. We could speed up the computation again by evaluating the constraint only on the current minibatch for each update, in which case we risk violating the fairness constraint.


For eq. (9), we can compute the gradient updates in all three methods with elementary linear algebra operations (matrix multiplications) and a single evaluation of the logistic function. While MPC is well suited for linear operations, most nonlinear functions are prohibitively expensive to evaluate in an MPC framework. Hence we tried two piecewise linear approximations for . The first was recently suggested for machine learning in an MPC context (Mohassel & Zhang, 2017) and is simply constant  and  for and respectively, and linear in between. The second uses the optimal first order Chebychev polynomial on each interval  for , and is constant  or  outside of  (Faiedh et al., 2001). While it is more accurate, we only report results for the simpler first approximation, as it yielded equal or better results in all our experiments.

As the largest number that can be represented in fixed-point format with  integer and  fractional bits is roughly , overflow becomes a common problem. Since we whiten the features  column-wise, we need to be careful whenever we add roughly  numbers or more, because we cannot even represent numbers greater than . In particular, the minibatch size has to be smaller than this limit. For large , the multiplication  in the fairness function  for the -rule is particularly problematic.

Hence, we split both factors into blocks of size with  and normalize the result of each blocked matrix multiplication by  before adding up the blocks. We then multiply the sum by . As long as , (and thus also ) can be represented with sufficient precision, which is the case in all our experiments, this procedure avoids under- and overflow. Note that we require the sample size  to be a multiple of . In practice, we have to either discard or duplicate part of the data. Since the latter may introduce bias, we recommend subsampling. Once we have (an approximation of) , we resort to normal matrix multiplication, as typically , see Table 1.

Division is prohibitively expensive in MPC. Hence, we set the minibatch size to a power of two, which allows us to use fast bit shifts for divisions when averaging over minibatches. To exploit the same trick when averaging over/across blocks in the blocked matrix multiplication, we choose  as the largest possible power of two, see Table 1. Algorithm 1 in Section B in the appendix describes the computations M and Reg have to run for fair model training using the Lagrangian multiplier technique and the -rule from eq. (9). We implicitly assume all computations are performed jointly on additively shared secrets.

5 Experiments

Adult Bank COMPAS German SQF
training examples
features 51 62 7 24 23
sensitive attr. 1 1 7 1 1
certification 802 ms 827 ms 288 ms 250 ms 765 ms
training 43 min 51 min 7 min 1 min 111 min
Table 1:

Dataset sizes and online timing results of MPC certification and training over 10 epochs with batch size 64.

The root cause for most technical difficulties pointed out in the previous section is the necessity to work with fixed-point numbers and the high computational cost of MPC. Hence, major concerns are loss of precision and infeasible running times. In this section, we show how to overcome both doubts and that fair training, certification and verification are feasible for realistic datasets.

5.1 Experimental Setup and Datasets

We work with two separate code bases. Our Python code does not implement MPC, to be able to flexibly switch between floating and fixed-point numbers as well as exact non-linear functions and their approximations. We use it mostly for validation and empirical guidance in our design choices. The full MPC protocol is implemented in C++ on top of the Obliv-C garbled circuits framework (Zahur & Evans, 2015a) and the Absentminded Crypto Kit (lib, ). This is done as described in Section 3 for the Lagrangian multiplier technique (see Section A in the appendix for more details). It accurately mirrors the computations performed by the first implementation on encrypted data.222Code is available at Except for the timing results in Table 1, all comparisons with floating-point numbers or non-linearities were done with the versatile Python implementation. Details about parameters and the algorithm can be found in Section B in the appendix.

Figure 2: Test set accuracy over the value for different optimization methods ( blue: iplb, orange: projected, green: Lagrange) and either no approximation (continuous) or a piecewise linear approximation (dashed) of the sigmoid using floating-point numbers. The gray dotted line is the baseline (see Section 4.1) and the black dashed line is unconstrained logistic regression (from scikit-learn).
Figure 3: The fraction of people with (continuous/dotted) and (dashed/dash-dotted) who get assigned positive outcomes (red: no approx. + float, purple: no approx. + fixed, yellow: pw linear + float, turquoise: pw linear + fixed, gray: baseline).

We consider 5 real world datasets, namely the adult (Adult), German credit (German), and bank market (Bank) datasets from the UCI machine learning repository (Lichman, 2013), the stop, question and frisk 2012 dataset (SQF),333 and the COMPAS dataset (Angwin et al., 2016) (COMPAS). For practical purposes (see Section 4), we subsample  examples from each dataset with the largest possible , see Table 1. Moreover, we also run on synthetic data, generated as described by Zafar et al. (2017c, Section 4.1), as it allows us to control the correlation between the sensitive attributes and the class labels. It is thus well suited to observe how different optimization techniques handle the fairness-accuracy trade off. For comparison we use the SLSQP approach described in Section 4.1 as a baseline. We run all methods for a range of constraint values in  and a corresponding range for SLSQP.

In the plots in this section, discontinuations of lines indicate failed experiments. The most common reasons are overflow and underflow for fixed-point numbers, and instability due to exploding gradients. Plots and analyses for the remaining datasets can be found in Section C in the appendix.

5.2 Comparing Optimization Techniques

First we evaluate which of the three optimization techniques works best in practice. Figure 2 shows the test set accuracy over the constraint value. By design, the synthetic dataset exhibits a clear trade-off between accuracy and fairness. The Lagrange technique closely follows the (dotted) baseline from (Zafar et al., 2017c), whereas iplb performs slightly worse (and fails for small ). Even though the projected gradient method formally satisfies the proxy constraint for the rule, it does so by merely shrinking the parameter vector , which is why it also fails for small . We analyze this behavior in more detail in Section C in the appendix.

The COMPAS dataset is the most challenging as it contains 7 sensitive attributes, one of which has only 10 positive instances in the training set. Since we enforce the fairness constraint individually for each sensitive attribute (we randomly picked one for visualization), the classifier tends to collapse to negative predictions. All three methods maintain close to optimal accuracy in the unconstrained region, but collapse more quickly than SLSQP. This example shows that the -rule proxy itself needs careful interpretation when applied to multiple sensitive attributes simultaneously and that our SGD based approach seems particularly prone to collapse in such a scenario. On the Bank dataset accuracy increases for iplb and Lagrange when the constraint becomes active as decreases until they match the baseline. Determining the cause of this—perhaps unintuitive—behavior requires further investigation. We currently suspect the constraint to act as a regularizer. The projected gradient method is unreliable on the Bank dataset.

Empirically, the Lagrangian multiplier technique is most robust with maximal deviations of accuracy from SLSQP of % across the 6 datasets and all constraint values. We substantiate this claim in Section C of the appendix. For the rest of this section we only report results for Lagrangian multipliers. Figure 2 also shows that using a piecewise linear approximation as described in Section 4 for the logistic function does not spoil performance.

5.3 Fair Training, Certification and Verification

Figure 3 shows how the fractions of users with positive outcomes in the two groups ( is continuous and is dashed) are gradually balanced as we decrease the fairness constraint . These plots can be interpreted as the degree to which disparate impact is mitigated as the constraint is tightened. The effect is most pronounced for the synthetic dataset by construction. As discussed above, the collapse for the COMPAS dataset occurs faster than for SLSQP due to the constraints from multiple sensitive attributes. In the Bank dataset, for large —before the constraint becomes active—the fractions of positive outcomes for differ, which is related to the slightly suboptimal accuracy at large that needs further investigation. However, as the constraint becomes active, the fractions are balanced at a similar rate as the baseline. Overall, our Lagrangian multiplier technique with fixed point numbers and piecewise linear approximations of non-linearities robustly manages to satisfy the p%-rule proxy at similar rates as the baseline with only minor losses in accuracy on all but the challenging COMPAS dataset.

In Table 1 we show the online running times of 10 training epochs on a laptop computer. While training takes several orders of magnitudes longer than a non-MPC implementation, our approach still remains feasible and realistic. We use the one time offline precomputation of multiplication triples described and timed in Mohassel & Zhang (2017, Table 2). As pointed out in Section 3, certification of a trained model requires checking whether . We already perform this check at least once for each gradient update during training. It only takes a negligible fraction of the computation time, see Table 1. Similarly, the operations required for certification stay well below one second.

Discussion. In this section, we have demonstrated the practicability of private and fair model training, certification and verification using MPC as described in Figure 1. Using the methods and tricks introduced in Section 4, we can overcome accuracy as well as over- and underflow concerns due to fixed-point numbers. Offline precomputation combined with a fast C++ implementation yield viable running times for reasonably large datasets on a laptop computer.

6 Conclusion

Real world fair learning has suffered from a dilemma: in order to enforce fairness, sensitive attributes must be examined; yet in many situations, users may feel uncomfortable in revealing these attributes, or modelers may be legally restricted in collecting and utilizing them. By introducing recent methods from MPC, and extending them to handle linear constraints as required for various notions of fairness, we have demonstrated that it is practical on real-world datasets to: (i) certify and sign a model as fair; (ii) learn a fair model; and (iii) verify that a fair-certified model has indeed been used; all while maintaining cryptographic privacy of all users’ sensitive attributes. Connecting concerns in privacy, algorithmic fairness and accountability, our proposal empowers regulators to provide better oversight, modelers to develop fair and private models, and users to retain control over data they consider highly sensitive.


The authors would like to thank Chris Russell and Phillipp Schoppmann for useful discussions and help with the implementation, as well as the anonymous reviewers for helpful comments. AG and MK were supported by The Alan Turing Institute under the EPSRC grant EP/N510129/1. MV was supported by EPSRC grant EP/M507970/1. AW acknowledges support from the David MacKay Newton research fellowship at Darwin College, The Alan Turing Institute under EPSRC grant EP/N510129/1 & TU/B/000074, and the Leverhulme Trust via the CFI.


  • (1) Absentminded crypto kit.
  • Al-Rubaie et al. (2017) Al-Rubaie, M., Wu, P. Y., Chang, J. M., and Kung, S. Privacy-preserving PCA on horizontally-partitioned data. In DSC, pp. 280–287. IEEE, 2017.
  • Angwin et al. (2016) Angwin, J., Larson, J., Mattu, S., and Kirchner, L. Machine bias: There is software used across the country to predict future criminals. and it is biased against blacks. ProPublica, May, 23, 2016.
  • Barocas & Selbst (2016) Barocas, S. and Selbst, A. D. Big data’s disparate impact. California Law Review, 104:671–732, 2016.
  • Bennett Capers (2012) Bennett Capers, I. Blind justice. Yale Journal of Law & Humanities, 24:179, 2012.
  • Biddle (2006) Biddle, D. Adverse impact and test validation: A practitioner’s guide to valid and defensible employment testing. Gower Publishing, Ltd., 2006.
  • Boyd & Vandenberghe (2004) Boyd, S. and Vandenberghe, L. Convex optimization. Cambridge university press, 2004.
  • Calders & Žliobaitė (2012) Calders, T. and Žliobaitė, I. Why unbiased computational processes can lead to discriminative decision procedures. In Discrimination and Privacy in the Information Society, pp. 43–59. Springer, 2012.
  • Damgård et al. (2012) Damgård, I., Pastro, V., Smart, N. P., and Zakarias, S. Multiparty computation from somewhat homomorphic encryption. In CRYPTO, volume 7417 of Lecture Notes in Computer Science, pp. 643–662. Springer, 2012.
  • Demmler et al. (2015a) Demmler, D., Dessouky, G., Koushanfar, F., Sadeghi, A., Schneider, T., and Zeitouni, S. Automated synthesis of optimized circuits for secure computation. In ACM Conference on Computer and Communications Security, pp. 1504–1517. ACM, 2015a.
  • Demmler et al. (2015b) Demmler, D., Schneider, T., and Zohner, M. ABY – a framework for efficient mixed-protocol secure two-party computation. In NDSS. The Internet Society, 2015b.
  • Dwork et al. (2012) Dwork, C., Hardt, M., Pitassi, T., Reingold, O., and Zemel, R. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, ITCS ’12, pp. 214–226. ACM, 2012.
  • Faiedh et al. (2001) Faiedh, H., Gafsi, Z., and Besbes, K.

    Digital hardware implementation of sigmoid function and its derivative for artificial neural networks.

    Proceeding of the 13th International Conference on Microelectronics, 2001., pp. 189 – 192, 11 2001.
  • Feldman et al. (2015) Feldman, M., Friedler, S., Moeller, J., Scheidegger, C., and Venkatasubramanian, S. Certifying and removing disparate impact. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 259–268, 2015.
  • Fredrikson et al. (2015) Fredrikson, M., Jha, S., and Ristenpart, T. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pp. 1322–1333, 2015.
  • Friedler et al. (2016) Friedler, S. A., Scheidegger, C., and Venkatasubramanian, S. On the (im)possibility of fairness. 2016. arXiv:1609.07236v1 [cs.CY].
  • Gascón et al. (2017) Gascón, A., Schoppmann, P., Balle, B., Raykova, M., Doerner, J., Zahur, S., and Evans, D.

    Privacy-Preserving Distributed Linear Regression on High-Dimensional Data.

    Proceedings on Privacy Enhancing Technologies, 2017(4):345–364, October 2017.
  • Goldreich (2004) Goldreich, O. The Foundations of Cryptography – Volume 2, Basic Applications. Cambridge University Press, 2004.
  • Goldreich et al. (1987) Goldreich, O., Micali, S., and Wigderson, A. How to play any mental game or A completeness theorem for protocols with honest majority. In STOC, pp. 218–229. ACM, 1987.
  • Graham (2017) Graham, C. NHS cyber attack: Everything you need to know about ’biggest ransomware’ offensive in history. Telegraph, May 20, 2017.
  • Grgić-Hlača et al. (2018) Grgić-Hlača, N., Zafar, M. B., Gummadi, K. P., and Weller, A.

    Beyond distributive fairness in algorithmic decision making: Feature selection for procedurally fair learning.

    In AAAI, 2018.
  • Handel et al. (2014) Handel, P., Skog, I., Wahlstrom, J., Bonawiede, F., Welch, R., Ohlsson, J., and Ohlsson, M. Insurance telematics: Opportunities and challenges with the smartphone solution. IEEE Intelligent Transportation Systems Magazine, 6(4):57–70, 2014.
  • Hardt et al. (2016) Hardt, M., Price, E., and Srebro, N.

    Equality of opportunity in supervised learning.

    In NIPS, 2016.
  • Juvekar et al. (2018) Juvekar, C., Vaikuntanathan, V., and Chandrakasan, A. Gazelle: A Low Latency Framework for Secure Neural Network Inference. IACR Cryptology ePrint Archive, 2018:73, 2018.
  • Keller et al. (2013) Keller, M., Scholl, P., and Smart, N. P. An architecture for practical actively secure MPC with dishonest majority. In ACM Conference on Computer and Communications Security, pp. 549–560. ACM, 2013.
  • Keller et al. (2017) Keller, M., Orsini, E., Rotaru, D., Scholl, P., Soria-Vazquez, E., and Vivek, S. Faster secure multi-party computation of AES and DES using lookup tables. In ACNS, volume 10355 of Lecture Notes in Computer Science, pp. 229–249. Springer, 2017.
  • Keller et al. (2018) Keller, M., Pastro, V., and Rotaru, D. Overdrive: Making SPDZ great again. In EUROCRYPT (3), volume 10822 of Lecture Notes in Computer Science, pp. 158–189. Springer, 2018.
  • Kroll et al. (2016) Kroll, J. A., Huey, J., Barocas, S., Felten, E. W., Reidenberg, J. R., Robinson, D. G., and Yu, H. Accountable algorithms. University of Pennsylvania Law Review, 165, 2016.
  • Lichman (2013) Lichman, M. UCI machine learning repository, 2013. URL
  • Lindell (2016) Lindell, Y. How To Simulate It – A Tutorial on the Simulation Proof Technique. IACR Cryptology ePrint Archive, 2016:46, 2016.
  • Liu et al. (2017) Liu, J., Juuti, M., Lu, Y., and Asokan, N. Oblivious neural network predictions via minionn transformations. In CCS, pp. 619–631. ACM, 2017.
  • Mohassel & Zhang (2017) Mohassel, P. and Zhang, Y. SecureML: A system for scalable privacy-preserving machine learning. In IEEE Symposium on Security and Privacy (SP), pp. 19–38, 2017.
  • Nikolaenko et al. (2013a) Nikolaenko, V., Ioannidis, S., Weinsberg, U., Joye, M., Taft, N., and Boneh, D. Privacy-preserving matrix factorization. In ACM Conference on Computer and Communications Security, pp. 801–812. ACM, 2013a.
  • Nikolaenko et al. (2013b) Nikolaenko, V., Weinsberg, U., Ioannidis, S., Joye, M., Boneh, D., and Taft, N.

    Privacy-preserving ridge regression on hundreds of millions of records.

    In IEEE Symposium on Security and Privacy, pp. 334–348. IEEE Computer Society, 2013b.
  • Schreurs et al. (2008) Schreurs, W., Hildebrandt, M., Kindt, E., and Vanfleteren, M. Cogitas, Ergo Sum. The Role of Data Protection Law and Non-discrimination Law in Group Profiling in the Private Sector. In Profiling the European Citizen, pp. 241–270. Springer, 2008.
  • Tramèr et al. (2016) Tramèr, F., Zhang, F., Juels, A., Reiter, M. K., and Ristenpart, T. Stealing machine learning models via prediction apis. In USENIX Security Symposium, pp. 601–618, 2016.
  • Veale & Binns (2017) Veale, M. and Binns, R. Fairer machine learning in the real world: Mitigating discrimination without collecting sensitive data. Big Data & Society, 4(2), 2017.
  • Veale & Edwards (2018) Veale, M. and Edwards, L. Clarity, Surprises, and Further Questions in the Article 29 Working Party Draft Guidance on Automated Decision-Making and Profiling. Computer Law & Security Review, 2018. doi: 10.1016/j.clsr.2017.12.002.
  • Yao (1986) Yao, A. C.-C. How to Generate and Exchange Secrets (Extended Abstract). In FOCS, pp. 162–167. IEEE Computer Society, 1986.
  • Zafar et al. (2017a) Zafar, M. B., Valera, I., Gomez-Rodriguez, M., and Gummadi, K. P. Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment. In WWW, 2017a.
  • Zafar et al. (2017b) Zafar, M. B., Valera, I., Rodriguez, M., Gummadi, K., and Weller, A. From parity to preference-based notions of fairness in classification. In Advances in Neural Information Processing Systems, pp. 228–238, 2017b.
  • Zafar et al. (2017c) Zafar, M. B., Valera, I., Rodriguez, M. G., and Gummadi, K. P. Fairness Constraints: Mechanisms for Fair Classification. In AISTATS, 2017c.
  • Zahur & Evans (2015a) Zahur, S. and Evans, D. Obliv-c: A language for extensible data-oblivious computation. IACR Cryptology ePrint Archive, 2015:1153, 2015a.
  • Zahur & Evans (2015b) Zahur, S. and Evans, D. Obliv-C: A Language for Extensible Data-Oblivious Computation. IACR Cryptology ePrint Archive, 2015:1153, 2015b.
  • Žliobaitė & Custers (2016) Žliobaitė, I. and Custers, B. Using sensitive personal data may be necessary for avoiding discrimination in data-driven decision models. Artificial Intelligence and Law, 24(2):183–201, 2016.

Appendix A Details of the MPC Protocols

Secret sharing.

A secret sharing scheme allows one to split a value (the secret) among two parties, so that no party has unilateral access to . In our setting, a user Alice will secret share a sensitive value, for example her race, among a modeler M and a regulator Reg. Several secret sharing schemes exist, including Shamir secret sharing, xor sharing, Yao sharing, or arithmetic multiplicative/additive sharing. In this work we alternate between Yao sharing and additive sharing for efficiency. In the latter, the value  is represented in a finite domain  with, for example . To share her race, Alice samples a value from  uniformly at random, and sends to M and  to Reg. We call each of and  a share, and denote them as  and . Now M and Reg can recover  by adding their shares, but each share on its own does not reveal anything about the value of  (other than that it is smaller than ). Note that the case where corresponds to xor sharing.

Function evaluation.

MPC can be classified in two groups depending on how  is represented: either as a Boolean or arithmetic circuit. All protocols proceed by having the parties jointly evaluate the circuit, processing it gate by gate. For each gate  for which the value for the input wires  is shared among the parties, the parties run a subprotocol to produce the value of the output wire, again shared, without revealing any information in the process. In the setting where we use arithmetic additive sharing, the two parties M and Reg hold shares, , and ,, respectively. In this case, is represented as an arithmetic circuit, and hence each gate in the circuit is either an addition or a multiplication. Note that if is an addition gate, then a sharing of can be obtained by having each party simply compute locally, i.e., without any interaction, , for . If is a multiplication, the subprotocol to compute shares of is much more costly. Fortunately, it can be divided into an offline and an online phase.

The preprocessing model in MPC.

In this model, two parties engage in an offline phase, which is data independent, and compute (and store) shared multiplication triples of the form , with . Here, are drawn uniformly at random, and each value is shared among the parties as explained above. In the online phase, a multiplication gate on shared values can be evaluated as follows: (1) each sets and , (2) the parties exchange their shares of and and reconstruct these values locally, and (3) each computes . The correctness of this protocol can be easily checked. Privacy relies on the uniform randomness of , and hence and completely mask the values of and , respectively. For a formal proof see (Demmler et al., 2015b).

Hence, for each multiplication in the function to be evaluated, the parties need to jointly generate a multiplication triple in advance. For computations with many multiplications (like in our case) this can be a costly process. However, this constraint is easy to accommodate in our architecture for private fair model training, as M and Reg can run the offline phase once “overnight”. Arithmetic multiplication via precomputed triples is a common technique, used in several popular MPC frameworks (Demmler et al., 2015b; Damgård et al., 2012). In this setting, several protocols for triple generation (which we did not describe) are available (Keller et al., 2018), and under continuous improvement. These protocols are often based on either Oblivious Transfer or Homomorphic Encryption.

The two-server model for multi-party learning.

Due to a sequence of theoretical and engineering breakthroughs, in the last three decades MPC has gone from being a mathematical curiosity to a technology of practical interest with commercial applications. Several generic protocols for MPC exists, such as the ones based on arithmetic sharing (Damgård et al., 2012), garbled circuits (Yao, 1986), or GMW (Goldreich et al., 1987), with several available implementations (Demmler et al., 2015b; Zahur & Evans, 2015b). These protocols have different trade-offs in terms of the number of parties they support, network requirements, and scalability for different kinds of computations. In our work, we focus on the -party case, as the MPC computation is done by M and Reg. The idea of privately outsourcing computation to two non-colluding parties in this way is recurrent in MPC, and often referred to as the two-server model (Mohassel & Zhang, 2017; Gascón et al., 2017; Nikolaenko et al., 2013b; Al-Rubaie et al., 2017).

While generic protocols exist, these do not yet scale to input sizes typically encountered in machine learning applications like ours. To circumvent this limitation, techniques tailored to specific applications have been proposed. Our protocols fall in this category, extending the SGD protocol from (Mohassel & Zhang, 2017), in which the following useful accelerating techniques are presented.

  • Efficient rescaling: As our arithmetic shares represent fixed-point numbers, we need to rescale by the precision after every multiplication. This involves dividing by , an expensive operation to do in MPC, and in particular in arithmetic sharing. Mohassel et al. show an elegant solution to this problem: the parties can rescale locally by dropping bits of their shares. It is not hard to see that this might produce the wrong result. However, the parameters of the arithmetic secret sharing scheme can be set such that with a tunable arbitrarily large probability the error is at most . This trick can be used for any division by a power of two.

  • Alternating sharing types: As already pointed out in previous work (Demmler et al., 2015b), alternating between secret sharing schemes can provide significant acceleration for some applications. Intuitively, arithmetic operations are fast in arithmetic shares, while comparisons are fast in schemes that represent functions as Boolean circuits. Examples of the latter are the GMW protocol and Yao’s garbled circuits. In our implementation, we follow this recipe and implement matrix-vector multiplication using arithmetic sharing, while for evaluating our variant of sigmoid, we rely on the protocol from (Mohassel & Zhang, 2017) implemented with garbled circuits using the Obliv-C framework (Zahur & Evans, 2015b).

  • Matrix multiplication triples: Another observation made by Mohassel et al. is that the idea described above for preprocessing multiplications over arithmetic shares can be reinterpreted at the level of matrices. This results in a faster online and offline phase (see (Mohassel & Zhang, 2017) for details).

How to prove that a protocol is secure.

We did not provide a formal definition of security in this paper, and instead referred the reader to (Mohassel & Zhang, 2017). In MPC, privacy in the case of semi-honest adversaries is argued in the simulation paradigm (see (Goldreich, 2004) or (Lindell, 2016) for formal definitions and detailed proofs). Intuitively, in this paradigm one proves that every inference that a party—in our case either Reg or M—could draw from observing the execution trace of the protocol could also be drawn from the output of the execution and the party’s input. This is done by proving the existence of a simulator that can produce an execution trace that is indistinguishable from the actual execution trace of the protocol. A crucial point is that the simulator only has access to the input and output of the party being simulated.

Appendix B Details of Fair Model Training

b.1 The Fair Training Algorithm

Algorithm 1 describes the computations M and Reg have to perform for fair model training using the Lagrangian multiplier technique and the -rule from eq. (9). In the next subsection we describe the parameter values. We implicitly assume all computations are performed jointly on additively shared secrets by M and Reg as described in Section 3. This means that M and Reg each receive a secret share of the protected attributes . Following the protocols outlined in Section 3, they can then jointly evaluate the steps in Algorithm 1. This allows them to operate on the sensitive values within the MPC computation, while preventing unilateral access to them by M and Reg. The result of these computations is the same as evaluating the algorithm as described with data in the clear.

BlockedMultShiftAvg stands for the blocked matrix multiplication to avoid overflow for fixed-point numbers described towards the end of Section 4. Note that it already contains the division by . The averaging within the blocked matrix multiplications as well as over the results thereof are done by fast bit shifts instead of slow MPC division circuits. This is possible, because we chose all parameters such that divisions are always by powers of two.

We found the piecewise linear approximation of the sigmoid function introduced in (Mohassel & Zhang, 2017)

to work best, see Figure 4.

Parties:  M, Reg.
Input:  (M)
Input:  (Reg) , ,
Input:  (Public) Learning rates , number of training examples , minibatch size , constraints , number of epochs .

1:  ,
3:  for all  from to  do
4:     for all  from to  do
13:     end for
14:  end for

Output:  Parameters 

Algorithm 1 Fair model training with private sensitive values using Lagrangian multipliers for .
Figure 4: Piecewise linear approximations for the non-linear sigmoid function (in black) from Mohassel & Zhang (2017) in blue and from Faiedh et al. (2001) in orange.

b.2 Description of Training Parameters

All our experiments use a batch size of 64, a fixed number of epochs scaling inversely with dataset size  (such that we always perform roughly 15 000 gradient updates), fixed learning rates of , and an annealing schedule for  in the interior point logarithmic barrier method as described by Boyd & Vandenberghe (2004). The weights for the gradients of the regular binary cross entropy loss (BCE) and the loss from the constraint terms (CON) follow the schedules

Weight decay, adaptive learning rate schedules. and momentum neither consistently improved nor impaired training. Therefore, all reported numbers were achieved with vanilla SGD, for fixed learning rates, and without any regularization. After extensive testing on all datasets, we converged to a fixed-point representation with 16 bits for the integer and fractional part respectively. The smaller the number of bits, the faster the MPC implementation and the higher the risk of loss of precision or over- and underflows. We found 16 bits to be the minimally needed precision for all our experiments to work.

Appendix C Additional Experimental Results

c.1 Results on Remaining Datasets

Analogously to Figures 2 and Figure 3 we report the results on test accuracy as well as the mitigation of disparate impact for the Lagrangian multiplier method in Figure 5. In the Adult dataset we are able to mitigate disparate impact with slightly worse accuracy as compared to the baseline. Note that the German dataset contains only 512 training and 200 test examples, which explains the discrete jumps in accuracy in minimal steps of . Hence, even though the Lagrangian multiplier technique here consistently removes disparate impact to a similar extent as the baseline, interpretations of results on such small datasets require great care. For the much larger stop, question and frisk dataset we again observe the curious initial increase in accuracy similar to our observations for the Bank dataset. In this dataset about of all samples have positive labels, which explains the near optimal accuracy when collapsing to always predict 1, which happens for the baseline as well for our method at a similar rate as decreases.

Figure 5: First row: The color code is blue: iplb, orange: projected, green: Lagrange with continuous lines for no approximation and dashed lines for piecewise linear approximation. The gray dotted line is the baseline and the dashed black line marks unconstrained logistic regression. Second row: Continuous/dotted lines correspond to and dashed/dash-dotted lines to . The color code is (red: no approx. + float, purple: no approx. + fixed, yellow: pw linear + float, turquoise: pw linear + fixed, gray: baseline).
Figure 6: We plot the fraction of people with (continuous/dotted) and with (dashed/dash-dotted) who get assigned positive outcomes over the constraint for 5 different datasets. The different colors correspond to (red: no approximation + floats, purple: no approximation + fixed-point, yellow: piecewise linear + floats, turquoise: piecewise linear + fixed-point, gray: baseline).

c.2 Disadvantages of Other Optimization Methods

In Section 5 we suggest the Lagrangian multiplier technique for fair model training using fixed-point numbers. Here we substantiate this suggestion with further empirical evidence. Figure 6 shows analogous results to Figure 3 and the second row of Figure 5. These plots reveal the shortcomings of the interior point logarithmic barrier and the projected gradient methods.

Interior Point Logarithmic Barrier method.

While the interior point logarithmic barrier method does balance the fractions of people being assigned positive outcomes between the two different demographic groups when the constraint is tightened, it soon breaks down entirely due to overflow and underflow errors. The number of failed runs was substantially higher than for the Lagrangian multiplier technique. As explained in (Boyd & Vandenberghe, 2004), when we increase the parameter  of the interior point logarithmic barrier method during training, the barrier becomes steeper, approaching the function

From this it becomes obvious that when facing tight constraints, the gradients might change from almost zero to extremely large values within a single update of the parameters . Moreover, iplb requires careful tuning and scheduling of . Hence, the interior point logarithmic barrier method, while achieving good results over some domains, is not well suited for MPC.

Projected gradient method.

In Figure 6, we observe that the projected gradient method seems to fail in most cases, since it does not actually balance the fractions of positive outcomes across the sensitive groups. There is a simple explanation why it can satisfy the constraint  for the -rule even with small  and still retain near optimal accuracy. Note that the accuracy only depends on the direction of , i.e., it is invariant to arbitrary rescaling of . Since the constraint  is always satisfied for , dividing any by a large enough factor will result in a classifier that achieves equal accuracy and satisfies the constraint (by continuity). However, minimizing the loss in the original logistic regression optimization problem (or equivalently maximizing the likelihood), which is not invariant under rescaling of , counteracts shrinking as it enforces high confidence of decisions, i.e., large . The projection method produces high accuracy classifiers with small weights that formally fulfill the fairness constraint, but do not properly mitigate disparate impact as measured by the true %-rule instead of the computational proxy. It also often fails for small constraint values, as the projection matrix in eq. (10) turns out to become near singular producing over- and underflow errors.

Appendix D Clarification of Privacy or Secrecy

In this work, privacy or secrecy constraints are separate from other theorized, setup-dependent attacks, e.g., model extraction (Tramèr et al., 2016) or inversion (Fredrikson et al., 2015). If relevant, modelers may need to consider these separately.