DeepAI
Log In Sign Up

System Cards for AI-Based Decision-Making for Public Policy

Decisions in public policy are increasingly being made or assisted by automated decision-making algorithms. Many of these algorithms process personal data for tasks such as predicting recidivism, assisting welfare decisions, identifying individuals using face recognition, and more. While potentially improving efficiency and effectiveness, such algorithms are not inherently free from issues such as bias, opaqueness, lack of explainability, maleficence, and the like. Given that the outcomes of these algorithms have significant impacts on individuals and society and are open to analysis and contestation after deployment, such issues must be accounted for before deployment. Formal audits are a way towards ensuring algorithms that are used in public policy meet the appropriate accountability standards. This work, based on an extensive analysis of the literature, proposes a unifying framework for system accountability benchmark for formal audits of artificial intelligence-based decision-aiding systems in public policy as well as system cards that serve as scorecards presenting the outcomes of such audits. The benchmark consists of 50 criteria organized within a four by four matrix consisting of the dimensions of (i) data, (ii) model, (iii) code, (iv) system and (a) development, (b) assessment, (c) mitigation, (d) assurance. Each criterion is described and discussed alongside a suggested measurement scale indicating whether the evaluations are to be performed by humans or computers and whether the evaluation outcomes are binary or on an ordinal scale. The proposed system accountability benchmark reflects the state-of-the-art developments for accountable systems, serves as a checklist for future algorithm audits, and paves the way for sequential work as future research.

READ FULL TEXT VIEW PDF

page 1

page 7

01/12/2022

The Human Factor in AI Safety

AI-based systems have been used widely across various industries for dif...
04/22/2022

"Public(s)-in-the-Loop": Facilitating Deliberation of Algorithmic Decisions in Contentious Public Policy Domains

This position paper offers a framework to think about how to better invo...
01/06/2023

Fiduciary Responsibility: Facilitating Public Trust in Automated Decision Making

Automated decision-making systems are being increasingly deployed and af...
03/22/2022

A Survey on Techniques for Identifying and Resolving Representation Bias in Data

The grand goal of data-driven decision-making is to help humans make dec...
03/07/2020

A Human-Centered Review of the Algorithms used within the U.S. Child Welfare System

The U.S. Child Welfare System (CWS) is charged with improving outcomes f...

1. Introduction

The use of AI-based systems in public policy has expanded dramatically in recent decades. They currently play an active part in informing policymakers in criminal justice decisions (Wexler, 2018), allocation of public resources (Gilman, 2020), public education decisions (Quay-de la Vallee and Duarte, 2019), and even national defense strategy (Mattis, 2018). Algorithms, defined broadly as a process or set of rules to be followed in problem-solving operations, have long been used in public policy in point systems, formal decision models, and informal rules of thumb (Allison and Halperin, 1972). For example, in 1978, the District Attorney’s office in Harris County, TX developed a scoring system for determining whether a defendant should be held in jail while awaiting trial, using seven questions (e.g., previous offenses, family in area, and home phone ownership)111Based on an interview with a former employee who worked on the development of the scale (March 16, 2021). Defendants who scored higher on the scale were considered less risky to release while awaiting trial than those who scored lower.

While algorithms have long been used in public policy, both the scope and complexity of the algorithms have been significantly increasing in recent years. The use of increasingly more complex algorithms that are usually based on artificial intelligence raises legal challenges in balancing the protection of privacy, civil liberties, and intellectual property; social challenges in creating trust and maintaining accountability; design challenges in producing systems that are both efficacious and transparent; and public policy challenges in incorporating algorithms into decision-making. The qualitatively and quantitatively different nature of the current wave of decision-aiding systems being introduced into public policy and society presents three challenges.

The first challenge is that with advances in data collection and machine learning, explicit algorithms have gained much broader acceptance than was previously possible. Most early research suggested that people were generally averse to computer algorithms, especially in human relations and moral decision-making

(Dawes, 1979; Dietvorst et al., 2015, 2018; Gogoll and Uhl, 2018; Lee, 2018). However, some research has suggested that this algorithm aversion has been dramatically declining (Kennedy et al., 2018; Logg, 2016, 2017). As trust in algorithms increases, so too does the danger of viewing decisions of these algorithms as ”just math” (Brayne, 2020), where even decisions that exacerbate racial bias and inequality are ”technowashed” by their encoding in an algorithm (Brayne, 2020; Winston and Burrington, 2018).

The second challenge is that the new wave of decision-aiding systems in public policy is often impenetrable and, thus, less prone to evaluation. The bail hearing algorithm mentioned above, despite its flaws, is reasonably easy to understand and, therefore, to challenge legally. Today’s algorithms are based on millions of data points, utilize hundreds of features, and model relationships using formulas that defy easy explanation. This makes them especially frustrating to challenge since, for instance, the exclusion of certain explicit factors like race can be accurately reconstructed from other factors in the model (O’Neil, 2016). Moreover, in part, because development teams for these algorithms tend to be quite homogeneous, the methods by which algorithms reconstruct racial and demographic categories may not be readily apparent to team members (Broussard, 2018). In more complex applications, such as facial or voice recognition, challenging a software’s assessment can be almost impossible.

The third challenge is that AI-based systems often obscure the policy decisions and social factors underlying their design and application. The large datasets on which the machine learning models are based were formed within the context of current social processes (Benjamin, 2019). Since the structure of machine learning problems involves using historical data to create future forecasts, biases inherent in social structures are reproduced in the algorithm (Brayne, 2020; Lambrecht and Tucker, 2019). It should not be surprising, for example, that predictive policing and sentencing algorithms are often accused of racial bias – since they are created using datasets produced by a racially biased criminal justice system, they reproduce those same patterns, even when the race is not explicitly used as a factor (Angwin et al., 2016). Especially in public policy, algorithms can produce devastating feedback loops. Pre-trial detention algorithms that (implicitly) incorporate information about a person’s income can result in people already at the margins of the economy losing their employment and falling further into poverty (O’Neil, 2016). Predictive policing algorithms that send more officers to areas with a large number of low-level, non-violent arrests are likely to result in still more low-level, non-violent arrests (Brayne, 2020). The process can quickly produce a negative feedback loop, punishing already disadvantaged members of society

On the other hand, the use of AI-based systems in decision-making is not entirely negative. For example, current sentencing and pre-trial detention algorithms have often been adopted by states looking to decrease overall prison populations without being labeled “soft on crime.” Indeed, Kleinberg et al. (2018) demonstrate that following the advice of an algorithm for evaluating defendant risk can reduce the number of people imprisoned by 42% without increasing crime rates or reduce crime rates by 24.8% without increasing prison populations. It must also be considered that the systems these algorithms are informing are not traditionally very accurate – many studies have found that professional judges are not particularly good at assessing defendant risk and are not very reliable in their judgments (Austin and Williams, 1977; Dhami and Ayton, 2001).

As AI-based decision-aiding systems expand into still more areas of life, the discussion needs to move beyond the arguments about whether the use of such systems is good or bad and towards a discussion of how those systems can be made accountable in terms of our ability to evaluate, challenge, and even override their behavior. However, to the best of our knowledge, a framework that serves as a comprehensive system accountability benchmark covering the whole lifecycle and different aspects of the system and its components is lacking in the literature. This study addresses this challenging gap. Our contribution is the proposal of a novel system accountability benchmark and resulting system cards for machine learning-based decision-aiding systems in public policy.

The rest of the work is structured as follows. Section 2 provides a review of selected literature and provides the background that drives the design and decisions for the proposed system accountability benchmark. Section 3 presents the criteria within the system accountability benchmark and its organization. Final remarks and directions for future research are provided in Section 4.

2. Background

There is an increasing number of studies that provide a framework or a discussion on algorithmic accountability including the relevant studies in algorithm auditing. Only a few of these studies have narrowed their focus to a single aspect or a dimension of algorithmic accountability. Many studies attempt to address different dimensions and aspects within an accountability framework in a holistic manner. On the other hand, as the field is still in development, no individual study captures all aspects and dimensions that are worthy of discussion for such a framework.

A group of studies (Loi and Spielkamp, 2021; Cech, 2021; Wieringa, 2020) largely concentrate on defining algorithmic accountability and what it means to be accountable for a decision-aiding system. Some studies (Henriksen et al., 2021; Slota et al., 2021) conduct interviews with relevant stakeholders to understand challenges around accountable artificial intelligence. Another group of studies (Raji et al., 2020; Mökander et al., 2021; Zicari et al., 2021) focuses on elaborating the process of auditing such as how an algorithm should be audited in terms of associated tasks, actions, and steps. Few studies (Bandy, 2021; Wilson et al., 2021) discuss actual applications of audits on algorithmic products. A relatively large number of studies (Supreme Audit Institutions of Finland, Germany, the Netherlands, Norway and the UK, 2021; Brown et al., 2021; Koshiyama et al., 2021; Naja et al., 2021; Tagiou et al., 2019; Krafft et al., 2021) are more extensive in scope and propose accountability frameworks that generally aim at capturing a diverse set of aspects and perspectives. The latter group also tends to present more tangible evaluation criteria rather than a relatively more abstract discussion of principles. On the other hand, some studies focus on particular themes such as nondiscrimination (Bartlett et al., 2020; Wilson et al., 2021; Saleiro et al., 2019), data (Bartlett et al., 2020; Gebru et al., 2021), models (Mitchell et al., 2019), transparency (Naja et al., 2021), human rights (McGregor et al., 2019), reputational concerns (Buhmann et al., 2020), audits by everyday users (Shen et al., 2021), and assurance (Batarseh et al., 2021; Kazim and Koshiyama, 2020). Finally, there are studies that examine regulations (Commission, 2021; Clarke, 2019; Office, 2020; Loi and Spielkamp, 2021; Kazim et al., 2021; MacCarthy, 2019).

Based on an analysis of the literature, several concepts that relate to an accountability framework are identified. The criteria within the framework that is presented in the next section are driven by these broad concepts: sociotechnical context, interpretability and explainability, transparency, nondiscrimination, robustness, and privacy and security.

A sociotechnical system refers to how human and technical elements connect with each other (Cooper and Foster, 1971). Computer systems including automated decision systems are not isolated and neutral tools but products of their sociotechnical context (Kroll, 2018; Dignum, 2017). The sociotechnical context of an algorithm includes identification of stakeholders encompassing but not limited to the people that might be adversely affected by the algorithm (Brown et al., 2021), the wider context that the algorithm is used in including intended uses and operators (Krafft et al., 2021) as well as the development phase of the system (Brown et al., 2021; Raji et al., 2020) as the sociotechnical context should be considered from the very beginning of the system design (Kroll, 2018). Human oversight, potential harms, and legal responsibility are necessarily discussed within the sociotechnical context (Slota et al., 2021). Existing sociotechnical issues or not accounting for the sociotechnical context may result in biased algorithms (Selbst et al., 2019; Mehrabi et al., 2021). Furthermore, ethics and regulations are attached to a systems’ use in its context rather than its mere technical specifications (Kroll, 2018; Brown et al., 2021; Maas, 2021). The proper balance when faced with tradeoff (e.g., privacy versus statistical accuracy, explainability versus intellectual property rights) also depends on the sociotechnical context in which the systems operate (Office, 2020).

Interpretability and explainability are often used interchangeably in the literature and their definitions, particularly the differences between the two, are not agreed upon. Accordingly, there are studies (Došilović et al., 2018; Gilpin et al., 2018; Marcinkevičs and Vogt, 2020; Lipton, 2018; Erasmus et al., 2021) recognizing the lack of rigor regarding their definitions and what it means to be explainable and interpretable. Gall (2018) states that interpretability is concerned with the ability to discern how the internal mechanics of the algorithm work and explainability is concerned with the ability to elucidate in human terms why and how these internal mechanics work the way they do. Accordingly, the next section distinguishes interpretability and explainability based on the target audience: model developers and ordinary human audience, respectively. Explainability and interpretability are indispensable characteristics of accountable systems as they make it possible or easier to identify and amend mistakes (Office, 2020), serve as a prerequisite for meaningful human oversight (Office, 2020), and help build trust in the system (Angelov et al., 2021; Longo et al., 2020). Various regulations already entitle users with the right to explanation (Selbst and Powles, 2017; Edwards and Veale, 2018; Barocas et al., 2020). There are different ways of providing explanations including individual-level versus population-level explanations, model-agnostic versus model-specific explanations (Kazim and Koshiyama, 2020; Koshiyama et al., 2021) and explanations that can be built inside the model or can be provided via supplemental tools (Office, 2020; Mökander et al., 2021). On the other hand, explainability and interpretability might have tradeoffs with statistical accuracy (Office, 2020; Angelov et al., 2021; Zicari et al., 2021) and privacy (Koshiyama et al., 2021). Although most discussion on explainability and interpretability concentrates on processing and post-processing stage (Koshiyama et al., 2021), the concept also covers the datasets (e.g., the need for data dictionaries and datasheets for datasets (Koshiyama et al., 2021)). In relation to assurance activities, a definition of AI assurance includes ensuring that the system outcomes are trustworthy and explainable (Batarseh et al., 2021). Moreover, algorithm insurance and certifications in the future might align closely with the systems’ explainability and interpretability (Kazim and Koshiyama, 2020; Koshiyama et al., 2021).

Transparency is used here as an umbrella term that refers to the openness associated with the decisions and actions during the lifecycle of the software, often manifesting itself in the form of clear, comprehensive, and useful documentation. Transparency does not necessarily result in explainable or interpretable models (Kroll, 2018) and it rather covers the explanations for the processes behind the model development (Selbst and Barocas, 2018). To begin with, transparency includes information about the system’s existence (Kroll, 2018; Brown et al., 2021) and reasons for its development and deployment (Kazim et al., 2021). It further includes the transparency about the model, its use, as well as the associated data collection process and how that data is used (Brown et al., 2021). Design transparency (e.g., code and data availability), documentation of design decisions, reproducibility, operational record keeping, impact assessments are considered as requirements of a transparent system (Kroll, 2021). It is important that transparency-ensuring information is available as a structured and preferably standardized documentation (Kroll, 2021) and covers the whole lifecycle of the software (Naja et al., 2021). Ensuring transparency brings harmful behavior to light (Weitzner et al., 2008), may help with detecting potential biases and other issues (Shen et al., 2021; Loi and Spielkamp, 2021), may improve the quality of the system (Loi and Spielkamp, 2021), and may enhance the trust in the system (Dignum, 2017). Transparency requires data, processes, and results to be ready for inspection and monitoring (Dignum, 2017). Such transparency can be direct via public transparency or indirect via transparency to auditors (Loi and Spielkamp, 2021). Regulatory bodies are also considering transparency as a principle for accountable and trustworthy AI systems (Felzmann et al., 2019). Lack of transparency may also result from the existing power dynamics and may reinforce the existing power imbalances (Kroll, 2018). Accordingly, transparency is a necessary condition for conducting internal or external audits (Raji et al., 2020) and for a meaningful accountability relation between the actor (the owner or operator of the decision-aiding system) and the forum (e.g., the general public, stakeholders, auditors) (Moss et al., 2021; Cech, 2021). On the other hand, transparency may conflict with proprietary rights (Slota et al., 2021; Crawford, 2015; Buhmann et al., 2020). It should be noted that transparency is not an end goal in itself but serves towards achieving accountability (Kroll, 2021; Tagiou et al., 2019).

Nondiscrimination is closely related to the concepts such as unbiasedness and fairness. Mehrabi et al. (2021) presents 23 types of bias, six types of discrimination, and 10 different definitions of fairness. The authors describe fairness as the lack of prejudice or preference towards a person or a group based on their inherent or acquired traits. Some fairness definitions are incompatible (Berk et al., 2021) and individual definitions cannot capture the full spectrum of notions of fairness and discrimination in legal, sociological, and philosophical contexts (Selbst et al., 2019). Therefore, fairness is usually employed as a vague term in the literature (McGregor et al., 2019). Due to this vagueness, some argue for focusing on unbiasedness rather than fairness (Bartlett et al., 2020). There are also differences between legal definitions and regulations regarding what is considered discrimination in different jurisdictions such as the EU and the US (Wachter et al., 2021). In this work, discrimination is used as a loose term referring to the legally or socially unacceptable or undesired treatment of different groups and individuals. Different jurisdictions and regulations provide legal protections for different predetermined demographic groups (e.g., based on gender and race) whereas the intersectional groups or excluded characteristics are not always necessarily protected (Wachter et al., 2021). Although fairness is usually observed in the model outcomes, to avoid discriminatory outcomes, the whole lifecycle of a system must be considered as design, development, and deployment are interlinked stages (McGregor et al., 2019). It also has strong implications on training datasets (Kazim and Koshiyama, 2020) as historic data from the imperfect world and unbalanced data sets may result in bias against historically marginalized and underrepresented groups, respectively (Mehrabi et al., 2021; Office, 2020). As moral values of the society regarding what is unfair evolves continuously (Kenward and Sinclair, 2021) and some potential discrimination might become known only after deployment, appropriate real-time fairness monitoring mechanisms are necessary (Office, 2020). On the other hand, there may be tradeoffs between fairness and privacy due to the processing of demographic data to measure fairness (Kazim et al., 2021) and between fairness and statistical accuracy (Office, 2020; Berk et al., 2021). Nevertheless, fairness is highly associated with trust in the system (Feuerriegel et al., 2020) and fairness of automated decision-aiding systems is increasingly finding a place in law and regulations (Hoffmann, 2019; Xiang and Raji, 2019; Hacker, 2018). The ethical solution to fairness can be achieved through creating processes and fora that are interactive and discursive (Buhmann et al., 2020), for which a tangible accountability framework can provide great support.

Robustness implies that automated decision-aiding systems must be testable during its lifecycle (Kroll, 2021), be safe against adversarial attacks aimed at manipulating the system (Kroll, 2021; Koshiyama et al., 2021; Wang et al., 2019; Sharma et al., 2020), and be statistically accurate (Koshiyama et al., 2021; Kazim and Koshiyama, 2020). Robustness also requires that the model performs well on the test set (Xu and Mannor, 2012) where the test data should have similar characteristics with the real-world cases. Accordingly, lack of robustness is associated with brittleness where a system fails in unexpected ways, particularly if subjected to situations that are different from what they are trained on (Jenkins et al., 2020). Therefore, robustness is closely linked with performance in the real world, which in turn is linked with the considerations given to the assumptions made during the development and the potentially unanticipated facts in the real world observed after deployment. Apart from the robustness of a system, a robust accountability evaluation requires the system to be accountable to an external forum (Moss et al., 2021), for which an objective accountability framework can be of assistance.

Privacy and security, in general, refer to the privacy and security of personal information but also include privacy of other critical data and security of the system against malicious actors (Office, 2020). The discussion around privacy should focus on the appropriate use of information rather than restricting access, particularly in the context of accountability (Weitzner et al., 2008). Security refers to jeopardizing a system’s confidentiality, integrity, or availability via intrusion (Strohmayer et al., 2021). Privacy and security are, then, also concerned with who are authorized to use the system (Brown et al., 2021) and should prevent personal or critical data leakage (Koshiyama et al., 2021). Privacy by design (Gürses et al., 2011) implies consideration of privacy requirements during the whole lifecycle of the system (Kroll, 2018) including the privacy concerns for the data which machine learning models are trained on (Office, 2020). Overall, appropriate documentation of the system and its components is necessary for inspecting the system for privacy and security concerns (Kroll, 2018). On the other hand, privacy may conflict with transparency (Weitzner et al., 2008). However, the right to privacy is recognized and enforced by many laws and regulations (Custers et al., 2018; Rustad and Koenig, 2019). To ensure privacy, in addition to adversarial testing (Raji et al., 2020) and privacy-enhancing techniques in such as differentially private machine learning (Gong et al., 2020) and federated learning (Li et al., 2020), there are mitigation strategies like anonymization (Koshiyama et al., 2021) and assurance procedures such as impact assessments related to data privacy and protection (Koshiyama et al., 2021; Bieker et al., 2016). Finally, security, in addition to following standard security principles and procedures during development, also requires the real-time monitoring of security risks in deployment (Kazim et al., 2021).

In summary, there is a plurality of perspectives and foci regarding algorithmic accountability and auditing algorithms. However, a unifying accountability benchmark is lacking in the literature. The next section presents the system accountability benchmark and system cards. System accountability benchmark is a framework for evaluating machine learning-based decision aiding systems with respect to relevant regulations, industry standards, and societal expectations. System cards are resulting scorecards when particular decision-aiding systems are subjected to evaluation based on the proposed system accountability benchmark.

3. System Accountability Benchmark and System Cards

System accountability benchmark consists of 50 criteria organized within a framework of four by four matrix. The rows of the matrix relate to the aspects of the software and the columns relate to the categories of the accountability criteria. The four aspects related to the software are Data, Model, Code, and System. The categories of the criteria are Development, Assessment, Mitigation, and Assurance. There are also four different evaluation classes (EC) to generate scores for each criterion: Binary by Human (BH), Binary by Machine (BM), Likert by Human (LH), and Likert by Machine (LM). Table 1 presents the proposed framework for system accountability benchmark.

Development # EC Assessment # EC Mitigation # EC Assurance # EC
Data Data Dictionary C111 LH Privacy, Data C211 BH Anonymization C311 BH Data Protection C411 LH
Datash., Collect. Proc. C112 LH Fairness, Data C212 BM Datash., Maint. C412 LH
Datash., Composition C113 LH Accuracy, Labels C213 LH Datash., Uses C413 LH
Datash., Motivation C114 LH Inspectability C214 LH
Datash., Preprocess. C115 LH
Model Reprod., Model C121 BM Interpretability C221 LH Advers., Training C321 BH Privacy, Model C421 BH
Design Transp., Model C122 LH Fairness, Model C222 LM Expl., Mitigation C322 BH Uses, Model C422 BH
Doc., Model C123 LH Testing, Advers. C223 LM Fairness, Mitigation C323 BH Doc., Capab. C423 LH
Privacy, Training C324 BH Explainability C424 LH
Code Reprod., Code C131 BM Privacy, Code C231 BH Review, Code C331 LH Certif., Developer C431 BH
Design Transp., Code C132 LH Security, Code C232 BH Diversity, Team C332 LM
Doc., Code C133 LH Testing Cards C233 BH
System Doc., Dev. C141 LH Awareness, Public C241 LH Monitor., Fairness C341 BH Record Keep., Oper. C441 BH
Risk, Humans C242 LH Monitor., Perf. C342 BH Uses, System C442 BH
Training, Operator C243 LH Oversight, Human C343 LH Doc., Acceptability C443 LH
Accuracy, System C244 LM Risk, Humans C344 LH Insurance C444 LH
Rating, Risk C445 LH
Table 1. System Accountability Benchmark Criteria

Data refers to the aspects related to the properties and characteristics of the data that the model learns from or works with. Model refers to the properties and behavior of the machine learning-based decision model that the system utilizes. Code refers to the actual source code underlying the system, including the code surrounding the decision model in its development and use. Finally, System refers to the software and its socio-technical context as a whole.

Documentation covers the criteria related to the managerial, operational, and informational record keeping for the development and use of the software. Assessment

covers the criteria that involve estimating the abilities or qualities of the system and its components.

Mitigation covers the criteria that can be utilized to prevent or mitigate potential shortcomings detected in Assessment. Finally, Assurance covers the criteria that aim to provide guarantees regarding the software and its use.

Each criterion has an associated method for evaluation. The evaluations can be performed either by humans manually or by machines automatically. The outcome of the evaluation can be either binary (i.e., inadequate or satisfactory) or scores on a Likert scale that is five-point (i.e., very poor, poor, average, good, excellent) or three-point (i.e., below, at, or above a reference level).

A system card is the overall outcome of the evaluation for a specific decision-aiding system. A system card is visualized as four concentric circles where each circle corresponds to a column in the framework for system accountability benchmark. Each criterion is denoted by an arc within its respective circle. The color of each arc denotes the evaluation outcome for the respective criteria, with the worst outcome denoted in red, the best outcome denoted in green, and other outcomes in the case of a Likert scale denoted with divergent colors between red and green. Figure 2 demonstrates the visualization of a system card.

Figure 2. Sample System Card

Four concentric circles where each circle corresponds to a column of the framework. Each criterion is denoted by an arc within its respective circle. The color of each arc denotes the evaluation outcome for the respective criteria, from worst (shown in red?) to best (shown in blue?).

In the remainder of this section, the accountability criteria belonging to each category are discussed.

3.1. Development

Data Dictionary criterion refers to the existence of data dictionaries. The data dictionary should cover training, validation, and test datasets as well as any other data that play a role in the development or use of the system. A data dictionary contains information on tables and their fields, typically including their meaning, source, relationship to other data, format, scale, and allowed values. (Uhrowczik, 1973; IBM, 1993). This criterion is examined by human experts and a score on the five-point Likert scale is assigned based on the existence, comprehensiveness, and quality of an efficacious data dictionary. The next four criteria are taken from the Datasheets proposed for datasets by Gebru et al. (2021). Collection Process aims to evoke information on how the data is collected and help other researchers collect similar data. Composition aims to describe the composition of the dataset including aspects related to its content including relationships, recommended data splits, and sensitive information. This criterion also includes the dataset size which is shown to be a powerful signal of assumed quality (Waggoner et al., 2019). Motivation aims to express the reasons for creating the dataset. Preprocessing aims to report on if and how the dataset is processed (e.g., cleaned, transformed, labeled). These four criteria are examined by human experts and scores on the five-point Likert scale are assigned based on the completeness and quality of appropriate documentation in line with the respective Datasheet specifications (Gebru et al., 2021).

Reproducibility, Model criterion refers to the existence of mechanisms that allow reproducibility of the model results. This criterion is evaluated by a machine and a binary outcome is assigned depending on whether the model results can be reproduced exactly. Design Transparency, Model criterion refers to the documentation of decisions and actions related to the design and development of the model such that an issue encountered in later stages can be traced to the specific decisions and actions as well as the actors responsible for those. Such documentation may also assist with future decision-making. Documentation, Model criterion refers to the existence and adequacy of documents describing the model, akin to the model cards proposed by Mitchell et al. (2019). These two criteria are examined by human experts and scores on the five-point Likert scale are assigned based on the availability and adequacy of appropriate documentation.

Reproducibility, Code criterion refers to the existence of mechanisms that allow the reproducibility of the code results. This criterion is evaluated by a machine and a binary outcome is assigned depending on whether the results from the code can be reproduced exactly. Design Transparency, Code criterion is similar to Design Transparency, Model, yet it focuses on decisions regarding the code design rather than the model. This criterion is measured by human experts and a score on the five-point Likert scale is assigned based on the availability and adequacy of relevant documentation. Documentation, Code criterion refers to the existence of a well-organized and well-documented codebase. The documentation should contain clear descriptions of the function of each piece of code, as well as information on the organization of the whole codebase and interrelations between different code pieces. Such documentation would decrease the risk of mistakes in development and increase the ease of developing improved or modified versions of the system. A score on the five-point Likert scale is assigned by human experts based on the conformity of the codebase design and the associated documentation to the industry standards.

Documentation, Development criterion refers to the existence of documents that describe and tailor the whole development lifecycle of the software from ideation and problem understanding to deployment and maintenance. It is examined by human experts and a score on the five-point Likert scale is assigned based on the availability and adequacy of appropriate documentation with respect to the industry standards.

3.2. Assessment

Privacy, Data criterion is concerned with respecting the data privacy regulations and best practices and not including personally identifiable information (PII) or information such that PII can be reconstructed except for the cases where relevant regulations such as GDPR (Union, 2016) allow it. This criterion is evaluated by human experts and a binary outcome is assigned. Fairness, Data criterion aims to ensure that the training data is free from bias. The input data should not contain any protected attribute (e.g., race, gender) as predictors unless it is of specific use to avoid discrimination against a protected group. The use of other variables that might serve as proxies to the protected attributes is subject to a statistical test for input data fairness. Given input variables, protected variables, and the target variable, Input Accountability Test (Bartlett et al., 2020) provides a statistical evaluation for this criterion based on the Civil Rights Act of 1964 (Congress, 2002). Therefore, this criterion is evaluated by a machine, and a binary outcome is assigned based on the test result. Accuracy, Labels criterion is concerned with whether the labels for individual data instances, particularly in the training set, are accurate and verified. Inspectability criterion refers to the availability of the infrastructure and tools to easily access and observe datasets that are employed or generated during the system’s development and use. Such tools should integrate data dictionary capabilities, complementing the data documentation criterion. These two criteria are examined by human experts and scores on the five-point Likert scale are assigned based on their independent investigation.

Interpretability criterion refers to whether the model allows developers and other technical experts to obtain explanations from it regarding how specific decisions are made. This criterion is examined by human experts and a score on the five-point Likert scale is assigned. Fairness, Model criterion is concerned with the potential discrimination in models’ outputs against protected groups (e.g., based on race or gender). There are multiple mutually exclusive formal criteria for fairness (Kleinberg et al., 2017; Barocas et al., 2019; Mehrabi et al., 2021). This work employs separation

that states that sensitive characteristics (e.g., race) must be statistically independent of the predicted label, given the ground truth label. In other words, for a binary classifier, it states that the false positive rates and false negative rates should be the same for different groups

(Barocas et al., 2019). Based on a machine’s evaluation of the separation measurement, a score is assigned on the five-point Likert scale. Testing, Adversarial is concerned with providing the model with intentionally designed inputs that aim to cause the model to make a mistake. For instance, in image classification tasks, a slight change that is sometimes invisible to immediate human perception, can cause those images to be misclassified. Another type of adversarial attack that is worthy of noting is data poisoning (Wang and Chaudhuri, 2018). Data poisoning occurs usually in systems that employ a continuous learning approach (i.e., the models that learn from the new data as they become available, possibly including the input-output pairs obtained through the model’s use in the real world). A malign actor can provide such data to a model, on purpose, to destabilize the model, worsen its performance, or manipulate it to produce certain outcomes for certain inputs. A model, therefore, must be tested against such adversarial attacks. Based on the results of the adversarial tests automatically performed by machines, a score is assigned on the five-point Likert scale.

Privacy, Code criterion refers to the respect for the confidentiality of user data and other sensitive data. The code should not allow for data leakage including those that can be obtained by reverse engineering (e.g., leakage of training data). Security, Code criterion refers to the overall security of the software against malicious attacks including those that aim to steal information, manipulate the software, or make the software unavailable. This criterion is closely related to information technology (IT) security. Therefore, the standards should be established with the involvement of experts in IT security. In practice, these two criteria are interwoven and are examined by human experts, and binary outcomes are assigned based on their independent investigations. Testing Cards criterion refers to whether there exist mechanisms and designs for testing the code. It should allow for testing the code’s behavior as a whole and the behavior of its individual parts. A testing card (e.g., test sheets (Atkinson et al., 2010)) should include information on the design of various tests (e.g., unit tests, system tests) and the results of those tests, as well as information on the code coverage. This criterion is examined by human experts and a binary outcome is assigned based on the existence of appropriately designed testing cards.

Awareness, Public is concerned with ensuring an appropriate level of familiarity and knowledge for the public in general and subjects of the decision-aiding system in particular about the existence, objectives, and mechanisms of the system. Such informed awareness enables the feedback and participation of the stakeholders regarding the use and potential modifications of the system. This criterion is examined by human experts and a score on the five-point Likert scale is assigned, possibly based on surveys conducted with the stakeholders and the relevant public. Risk, Humans criterion is concerned with the potential risks of the deployed software and its use on the rights and freedoms of individuals. At a bare minimum, the impacts of the software must be contemplated with potential risks to those rights defined in the Universal Declaration of Human Rights (United Nations, 1948) including but not limited to the right to life, liberty and personal security; rights to equality, marriage and family, education, fair public hearing; and freedom from arbitrary arrest. This criterion is examined by human experts and a score on the five-point Likert scale is assigned based on potential risks on human rights. A decision-aiding system that clearly violates any human rights cannot satisfy this criterion. Training, Operator criterion is interested in whether the operators of the system have received adequate training on the nature and limitations of the model. It follows that such training, possibly tailored towards different use cases, should be available alongside the software. This criterion is examined by human experts and a score on the five-point Likert scale is assigned based on the existence and adequateness of the training. Accuracy, System criterion refers to the appropriateness of employed accuracy metrics in evaluating a systems’ performance, whether the evaluation is performed on a suitable test set, as well as the acceptability of the accuracy level. A metric that incorporates the rate of false positives, especially if the model is followed by considerable real-world implications, should be considered. To this end, receiver operating characteristic (ROC) curves are a suitable technique to demonstrate the model’s predictive abilities. This criterion is examined by a machine and a score in the 3-point Likert scale is assigned based on whether the system performs below, at, or above a reference level.

3.3. Mitigation

Anonymization is a mitigation strategy that removes personally identifiable information and makes the data compliant with privacy regulations and best practices. It is examined by human experts and a binary outcome is assigned based on the need for and the existence of anonymization mechanisms.

Adversarial, Training is a mitigation technique especially when the score obtained for the Testing, Adversarial criterion is not satisfactory. With adversarial training (Wang et al., 2019), the adversarial samples are learned by the model in the training stage so the model becomes more robust against adversarial attacks. Explanations, Mitigation criterion refers to the utilization of tools to provide explanations in case the original model does not have inherent mechanisms for providing explanations. For instance, surrogate explanations (Ribeiro et al., 2016; Lundberg and Lee, 2017) aim to explain the more complex models by training simpler models on the original input-output pairs. Fairness, Mitigation criterion refers to the utilization of techniques in pre-processing, in-processing, or post-processing stages to ensure fairness in the model outcomes (Mehrabi et al., 2021). Privacy, Training criterion refers to the utilization of privacy-preserving machine learning techniques such as federated learning (Li et al., 2020) and differential privacy (Gong et al., 2020). These four criteria are examined by human experts and binary outcomes are assigned based on the need for and the existence of respective mitigation mechanisms.

Review, Code criterion refers to the employment of code review practices during the development. This criterion is examined by human experts and a score on the five-point Likert scale is assigned based on the existence, quality, and application of well-established code review practices that include code reviewers other than the original authors of the code. Diversity, Team criterion refers to the diversity of the developer team. This work employs social category diversity based on the representation of diverse demographic groups (Liang et al., 2007). The diversity is computed by machines using entropy (Teachman, 1980) and a score is assigned on the five-point Likert scale.

Monitoring, Fairness and Monitoring, Performance refer to the existence of infrastructure and mechanisms to monitor the fairness metrics and the accuracy of the ML model, respectively, in its real-world use. These two criteria are examined by human experts and binary outcomes are assigned based on the availability of appropriate monitoring mechanisms. Oversight, Human criterion refers to the principle that the existence of automated systems should not reduce the power and responsibility of humans either in explicit or implicit ways (McGregor et al., 2019). There are different levels of human involvement (i.e., human-in-the-loop where human consent is necessary by default, human-on-the-loop where humans can overturn the outcome of a decision-aiding system, and human-out-of-the-loop where no human oversight exists (Wieringa, 2020)). A human-out-of-the-loop system cannot satisfy this criterion for any type of application. For higher-risk applications, humans should oversee the algorithmic decisions and modify them as appropriate and necessary. As the risk increases, the necessary human involvement should increase. However, even low-risk applications should have a mechanism where humans can overrule the decision made by the system in line with appropriate policy guidelines. This criterion is examined by human experts and a score on the five-point Likert scale is assigned based on the existence and appropriate design of the policies and necessary mechanisms for human involvement. Risk, Humans criterion aims to ensure that, even in the absence of any negligence or bad faith, appropriate remedies are provided for unintended or unexpected harms. If and when such harms occur, appropriate policies and mechanisms should be readily available to establish the nature of remedies with respect to the seriousness of the harm. This criterion is examined by human experts and a score on the five-point Likert scale is assigned based on the availability, sufficiency, and effectiveness of the remedies.

3.4. Assurance

Data Protection criterion is concerned with whether data protection impact assessments (Office, 2021; Bieker et al., 2016) are prepared for the system. This criterion is examined by human experts and a score on the five-point Likert scale is assigned based on the availability and adequacy of impact assessments with respect to relevant regulations and best practices. Datasheets, Maintenance criterion aims to ensure dataset maintenance plans are in place and the plan allows for future communication regarding the dataset. Datasheets, Uses aims to clarify tasks for which the dataset can be used as well as the tasks that the dataset should not be used for. These two criteria are examined by human experts and scores on the five-point Likert scale are assigned based on the completeness and quality of appropriate documentation in line with the respective Datasheet specifications (Gebru et al., 2021).

Privacy, Model criterion refers to whether appropriate privacy precautions are in place with respect to the model architecture and behavior. Uses, Model criterion refers to the alignment of the model’s actual use in the real world to the purpose that it is originally intended for. These two criteria are examined by human experts and binary outcomes are assigned. Documentation, Capabilities criterion refers to the existence and clarity of the documents that provide detailed information on the capabilities of the model (e.g., tasks that it can successfully perform, conditions that are necessary for it to perform). It is also important that the documentation explicitly describes the tasks and situations for which the model is not suitable to use (i.e., its limitations). Explainability criterion refers to the ease of understanding and interpretation for the ordinary human audience regarding how the model arrives at its decisions. A model should be self-explanatory or should provide additional tools for explaining how it produces its outputs, both for the general model behavior and for individual cases. Explainability cannot be satisfied merely by explicating the calculations performed by the model but requires the audience to be able to make counterfactual claims about the model’s behavior (Wang et al., 2021). These two criteria are examined by human experts and scores on the five-point Likert scale are assigned.

Certification, Developer criterion refers to whether developers of the system (i.e., engineers, designers, testers). have appropriate and necessary certificates. This criterion is examined by human experts and a binary outcome is assigned.

Record Keeping, Operational

criterion refers to the existence of technological infrastructures that provides logging capabilities for the duration of the system’s development and use. The records must clearly indicate inputs, outputs, model files as well as information on the use of the system such as timestamps and operators. The records must be complete and immutable and must be stored for 10 years, covering, for instance, the log retention periods of six years as inscribed by the Health Insurance Probability and Accountability Act

(Congress, 1996) for medical records, seven years as inscribed by the Sarbanes-Oxley Act (Congress, 2002) for corporate audit records, seven years as inscribed for Texas State Agencies and Public Universities (Commission, 2020) for audit records, and 10 years as inscribed by Oregon Secretary of State (State, ) for administrative rule records. This criterion is examined by human experts and a binary outcome is assigned based on the existence and conformity of the infrastructure and procedures for record keeping. Uses, System criterion refers to the alignment of the system’s actual use in the real world to the purpose that the system is originally intended for. This criterion is examined by human experts and a binary outcome is assigned. Documentation, Acceptability criterion refers to the existence and completeness of the documents that elaborate and certify the acceptance criteria for the system and the conditions necessary to satisfy those criteria. A set of fixed acceptance criteria is neither established in the literature nor proposed in this work. However, the criteria should be unambiguous, consistent, and comprehensive (i.e., cover relevant aspects related to functionality, performance, interface, security, and software safety (Wallace and Cherniavsky, 1990)). The criteria should be developed with the involvement of key stakeholders and must account for the project requirements (Modeling and Enterprise, 2011). This criterion is examined by human experts and a score on the five-point Likert scale is assigned based on the availability and adequacy of appropriate documentation. Insurance criterion is concerned with whether the system is insured for liability. This criterion is examined by human experts and a score on the five-point Likert scale is assigned based on the existence and coverage of the insurance. Rating, Risk criterion is concerned with risk ratings assigned to the system by independent risk agencies, as some regulations (e.g., GDPR (Union, 2016) and Artificial Intelligence Act (Commission, 2021)) have a tiered system of expectations based on the risk involved with the technology. This criterion is examined by human experts (e.g., risk rating agencies) and a score on the five-point Likert scale is assigned.

4. Conclusion

The contribution of this work is the proposed system accountability benchmark that produces system cards for machine learning-based decision-aiding systems in various public policy functions. The entries of a system card are to be filled by relevant internal or external auditors based on a proper evaluation of the overall system and its components. Apart from serving the auditing purposes, the framework is intended to be considered during the whole lifecycle of the system to assist its design, development, and deployment in accordance with accountability principles.

Considering the evolving and cumulative nature of science in general and the infancy of the algorithmic accountability field in particular, the proposed framework for system accountability benchmark is not suggested to be the ultimate solution for all and at once. However, being a tangible and relatively specific framework, the system accountability benchmark and system cards contribute to moving the current efforts forward towards real-world impact, ignite further discussions on the topic, and motivate holding decision-aiding systems in public policy accountable in practice.

The system accountability benchmark paves the way for three major lines of future work that are also sequentially linked. First, further research is required for establishing specific and full-fledged guidelines or automated tools to evaluate individual criteria proposed in the framework. The second line of work would be towards developing mature auditing procedures that elaborate on what to audit, when to audit, who audits, and how audits must be conducted. Finally, in light of the auditing procedure, the proposed framework alongside the specific guidelines and automated tools can be applied to generate system cards for different AI-based decision-aiding systems in diverse public policy functions across various governing bodies.

Acknowlegments

The authors would like to thank Drs. Andrew C. Michaels, Ryan Kennedy, and Lydia B. Tiede for their insightful comments on an earlier draft of this manuscript. This material is based upon work supported by the National Science Foundation under Grant CCF-2131504. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

References

  • G. T. Allison and M. H. Halperin (1972) Bureaucratic Politics: A Paradigm and Some Policy Implications. World Politics 24, pp. 40–79. External Links: Document, ISSN 0043-8871, Link Cited by: §1.
  • P. P. Angelov, E. A. Soares, R. Jiang, N. I. Arnold, and P. M. Atkinson (2021) Explainable artificial intelligence: an analytical review. WIREs Data Mining and Knowledge Discovery 11 (5), pp. e1424. External Links: Document Cited by: §2.
  • J. Angwin, J. Larson, L. Kirchner, and S. Mattu (2016) Machine bias. External Links: Link Cited by: §1.
  • C. Atkinson, F. Barth, and D. Brenner (2010) Software Testing Using Test Sheets. In Proc. International Conference on Software Testing, Verification, and Validation Workshops, pp. 454–459. External Links: Document Cited by: §3.2.
  • W. Austin and T. A. Williams (1977) A Survey of Judges’ Responses to Simulated Legal Cases: Research Note on Sentencing Disparity. The Journal of Criminal Law and Criminology (1973-) 68 (2), pp. 306–310. External Links: Document, ISSN 0091-4169, Link Cited by: §1.
  • J. Bandy (2021) Problematic Machine Behavior: A Systematic Literature Review of Algorithm Audits. Proc. ACM Conference on Human-Computer Interaction, pp. 74:1–74:34. External Links: Document, Link Cited by: §2.
  • S. Barocas, M. Hardt, and A. Narayanan (2019) Fairness and Machine Learning. fairmlbook.org. Cited by: §3.2.
  • S. Barocas, A. D. Selbst, and M. Raghavan (2020) The hidden assumptions behind counterfactual explanations and principal reasons. In Proc. ACM Conference on Fairness, Accountability, and Transparency, pp. 80–89. External Links: Document, ISBN 978-1-4503-6936-7, Link Cited by: §2.
  • R. P. Bartlett, A. Morse, N. Wallace, and R. Stanton (2020) Algorithmic Discrimination and Input Accountability under the Civil Rights Acts. Social Science Research Network (english). External Links: Link Cited by: §2, §2, §3.2.
  • F. A. Batarseh, L. Freeman, and C. Huang (2021) A survey on artificial intelligence assurance. Journal of Big Data 8 (1), pp. 60. External Links: Document, ISSN 2196-1115, Link Cited by: §2, §2.
  • R. Benjamin (2019) Race After Technology: Abolitionist Tools for the New Jim Code. Polity Press, Medford, MA. External Links: ISBN 978-1-5095-2643-7 Cited by: §1.
  • R. Berk, H. Heidari, S. Jabbari, M. Kearns, and A. Roth (2021) Fairness in Criminal Justice Risk Assessments: The State of the Art. Sociological Methods & Research 50 (1), pp. 3–44 (english). External Links: Document, ISSN 0049-1241, Link Cited by: §2.
  • F. Bieker, M. Friedewald, M. Hansen, H. Obersteller, and M. Rost (2016) A process for data protection impact assessment under the european general data protection regulation. In Proc. Privacy Technologies and Policy: Annual Privacy Forum, pp. 21–37. External Links: Document, ISBN 978-3-319-44760-5 Cited by: §2, §3.4.
  • S. Brayne (2020) Predict and Surveil: Data, Discretion, and the Future of Policing. Oxford University Press, New York, NY (english). External Links: ISBN 978-0-19-068409-9 Cited by: §1, §1.
  • M. Broussard (2018) Artificial Unintelligence: How Computers Misunderstand the World. The MIT Press, Cambridge, MA (english). External Links: ISBN 978-0-262-03800-3 Cited by: §1.
  • S. Brown, J. Davidovic, and A. Hasan (2021) The algorithm audit: Scoring the algorithms that score us. Big Data & Society 8 (1), pp. 2053951720983865 (english). External Links: Document, ISSN 2053-9517, Link Cited by: §2, §2, §2, §2.
  • A. Buhmann, J. Paßmann, and C. Fieseler (2020) Managing Algorithmic Accountability: Balancing Reputational Concerns, Engagement Strategies, and the Potential of Rational Discourse. Journal of Business Ethics 163 (2), pp. 265–280 (english). External Links: Document, ISSN 1573-0697, Link Cited by: §2, §2, §2.
  • F. Cech (2021) The agency of the forum: Mechanisms for algorithmic accountability through the lens of agency. Journal of Responsible Technology 7-8, pp. 100015. External Links: Document, ISSN 2666-6596, Link Cited by: §2, §2.
  • Y. D. Clarke (2019) Algorithmic Accountability Act of 2019. (english). External Links: Link Cited by: §2.
  • E. Commission (2021) Proposal for a Regulation of the European Parliament and of the Council Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) and Amending Certain Union Legislative Acts. (english). External Links: Link Cited by: §2, §3.4.
  • T. S. L. a. A. Commission (2020) Retention Schedules for Texas State Agencies and Public Universities. External Links: Link Cited by: §3.4.
  • U. S. Congress (1996) Health Insurance Portability and Accountability Act of 1996. (english). External Links: Link Cited by: §3.4.
  • U. S. Congress (2002) Corporate Fraud Accountability Act of 2002. (english). External Links: Link Cited by: §3.2, §3.4.
  • R. Cooper and M. Foster (1971) Sociotechnical systems. American Psychologist 26 (5), pp. 467–474. External Links: Document, ISSN 1935-990X Cited by: §2.
  • K. Crawford (2015) Can an algorithm be agonistic? Ten scenes from life in calculated publics. Science, Technology, & Human Values 41 (1), pp. 77–92. External Links: Document, Link Cited by: §2.
  • B. Custers, F. Dechesne, A. M. Sears, T. Tani, and S. van der Hof (2018) A comparison of data protection legislation and policies across the EU. Computer Law & Security Review 34 (2), pp. 234–243 (english). External Links: Document, ISSN 0267-3649, Link Cited by: §2.
  • R. M. Dawes (1979) The robust beauty of improper linear models in decision making. American Psychologist 34 (7), pp. 571–582. External Links: Document Cited by: §1.
  • M. K. Dhami and P. Ayton (2001) Bailing and jailing the fast and frugal way. Journal of Behavioral Decision Making 14 (2), pp. 141–168 (english). External Links: Document, ISSN 1099-0771, Link Cited by: §1.
  • B. J. Dietvorst, J. P. Simmons, and C. Massey (2018) Overcoming algorithm aversion: People will use imperfect algorithms if they can (even slightly) modify them. Management Science Series A-theory 64 (3), pp. 1155–1170. External Links: Document Cited by: §1.
  • B. J. Dietvorst, J. P. Simmons, and C. Massey (2015) Algorithm aversion: People erroneously avoid algorithms after seeing them err.. Journal of Experimental Psychology: General 144 (1), pp. 114–126. External Links: Document, Link Cited by: §1.
  • V. Dignum (2017) Responsible autonomy. In Proc. International Joint Conference on Artificial Intelligence, pp. 4698–4704. External Links: Document Cited by: §2, §2.
  • F. K. Došilović, M. Brčić, and N. Hlupić (2018) Explainable artificial intelligence: A survey. In Proc. International Convention on Information and Communication Technology, Electronics and Microelectronics, pp. 0210–0215. External Links: Document Cited by: §2.
  • L. Edwards and M. Veale (2018) Enslaving the Algorithm: From a “Right to an Explanation” to a “Right to Better Decisions”?. IEEE Security and Privacy 16 (3), pp. 46–54. External Links: Document, ISSN 1558-4046 Cited by: §2.
  • A. Erasmus, T. D. P. Brunet, and E. Fisher (2021) What is Interpretability?. Philosophy & Technology 34 (4), pp. 833–862 (english). External Links: Document, ISSN 2210-5441, Link Cited by: §2.
  • H. Felzmann, E. F. Villaronga, C. Lutz, and A. Tamò-Larrieux (2019) Transparency you can trust: Transparency requirements for artificial intelligence between legal norms and contextual concerns. Big Data & Society 6 (1), pp. 2053951719860542 (english). External Links: Document, ISSN 2053-9517, Link Cited by: §2.
  • S. Feuerriegel, M. Dolata, and G. Schwabe (2020) Fair AI. Business & Information Systems Engineering 62 (4), pp. 379–384 (english). External Links: Document, ISSN 1867-0202, Link Cited by: §2.
  • R. Gall (2018) Machine Learning Explainability vs Interpretability: Two concepts that could help restore trust in AI. (en-US). External Links: Link Cited by: §2.
  • T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. D. III, and K. Crawford (2021) Datasheets for datasets. Communications of The ACM 64 (12), pp. 86–92. External Links: Document, ISSN 0001-0782, Link Cited by: §2, §3.1, §3.4.
  • M. E. Gilman (2020) Poverty Lawgorithms: A Poverty Lawyer’s Guide to Fighting Automated Decision-Making Harms on Low-Income Communities. Social Science Research Network (english). External Links: Link Cited by: §1.
  • L. H. Gilpin, D. Bau, B. Z. Yuan, A. Bajwa, M. Specter, and L. Kagal (2018) Explaining Explanations: An Overview of Interpretability of Machine Learning. In

    Proc. IEEE International Conference on Data Science and Advanced Analytics

    ,
    pp. 80–89. External Links: Document Cited by: §2.
  • J. Gogoll and M. Uhl (2018) Rage against the machine: Automation in the moral domain. Journal of Behavioral and Experimental Economics 74, pp. 97–103. External Links: Document, ISSN 2214-8043, Link Cited by: §1.
  • M. Gong, Y. Xie, K. Pan, K. Feng, and A.K. Qin (2020) A Survey on Differentially Private Machine Learning. IEEE Computational Intelligence Magazine 15 (2), pp. 49–64. External Links: Document, ISSN 1556-6048 Cited by: §2, §3.3.
  • S. Gürses, C. Troncoso, and C. Diaz (2011) Engineering privacy by design. (english). External Links: Link Cited by: §2.
  • P. Hacker (2018) Teaching fairness to artificial intelligence: Existing and novel strategies against algorithmic discrimination under EU law. Common Market Law Review, pp. 1143–1185. External Links: ISSN 0165-0750, Link Cited by: §2.
  • A. Henriksen, S. Enni, and A. Bechmann (2021) Situated Accountability: Ethical Principles, Certification Standards, and Explanation Methods in Applied AI. In Proc. AAAI/ACM Conference on AI, Ethics, and Society, pp. 574–585. External Links: Document, ISBN 978-1-4503-8473-5, Link Cited by: §2.
  • A. L. Hoffmann (2019) Where fairness fails: data, algorithms, and the limits of antidiscrimination discourse. Information, Communication & Society 22 (7), pp. 900–915. External Links: Document, ISSN 1369-118X, Link Cited by: §2.
  • IBM (1993) IBM Dictionary of Computing. McGraw-Hill, New York, NY. External Links: ISBN 978-0-07-031488-7 Cited by: §3.1.
  • O. C. Jenkins, D. Lopresti, and M. Mitchell (2020) Next wave artificial intelligence: Robust, explainable, adaptable, ethical, and accountable. External Links: 2012.06058 Cited by: §2.
  • E. Kazim, D. M. T. Denny, and A. Koshiyama (2021) AI auditing and impact assessment: according to the UK information commissioner’s office. AI and Ethics 1 (3), pp. 301–310 (english). External Links: Document, ISSN 2730-5961, Link Cited by: §2, §2, §2, §2.
  • E. Kazim and A. Koshiyama (2020) AI Assurance Processes. Social Science Research Network (english). External Links: Link Cited by: §2, §2, §2, §2.
  • R. Kennedy, P. Waggoner, and M. Ward (2018) Trust in Public Policy Algorithms. SSRN Scholarly Paper. External Links: Link Cited by: §1.
  • B. Kenward and T. Sinclair (2021) Machine morality, moral progress, and the looming environmental disaster. Cognitive Computation and Systems 3 (2), pp. 83–90. External Links: Document, Link Cited by: §2.
  • J. Kleinberg, H. Lakkaraju, J. Leskovec, J. Ludwig, and S. Mullainathan (2018) Human Decisions and Machine Predictions. The Quarterly Journal of Economics 133 (1), pp. 237–293. External Links: Document, ISSN 0033-5533, Link Cited by: §1.
  • J. Kleinberg, S. Mullainathan, and M. Raghavan (2017) Inherent trade-offs in the fair determination of risk scores. In Proc. Innovations in Theoretical Computer Science Conference, pp. 43:1–43:23. External Links: Document, ISBN 978-3-95977-029-3 Cited by: §3.2.
  • A. Koshiyama, E. Kazim, P. Treleaven, P. Rai, L. Szpruch, G. Pavey, G. Ahamat, F. Leutner, R. Goebel, A. Knight, J. Adams, C. Hitrova, J. Barnett, P. Nachev, D. Barber, T. Chamorro-Premuzic, K. Klemmer, M. Gregorovic, S. Khan, and E. Lomas (2021) Towards Algorithm Auditing: A Survey on Managing Legal, Ethical and Technological Risks of AI, ML and Associated Algorithms. Social Science Research Network (english). External Links: Link Cited by: §2, §2, §2, §2.
  • P. M. Krafft, M. Young, M. Katell, J. E. Lee, S. Narayan, M. Epstein, D. Dailey, B. Herman, A. Tam, V. Guetler, C. Bintz, D. Raz, P. O. Jobe, F. Putz, B. Robick, and B. Barghouti (2021) An Action-Oriented AI Policy Toolkit for Technology Audits by Community Advocates and Activists. In Proc. ACM Conference on Fairness, Accountability, and Transparency, pp. 772–781. External Links: Document, ISBN 978-1-4503-8309-7, Link Cited by: §2, §2.
  • J. A. Kroll (2018) The fallacy of inscrutability. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 376 (2133), pp. 20180084. External Links: Document, Link Cited by: §2, §2, §2.
  • J. A. Kroll (2021) Outlining Traceability: A Principle for Operationalizing Accountability in Computing Systems. In Proc. ACM Conference on Fairness, Accountability, and Transparency, pp. 758–771. External Links: Document, ISBN 978-1-4503-8309-7, Link Cited by: §2, §2.
  • A. Lambrecht and C. Tucker (2019) Algorithmic Bias? An Empirical Study of Apparent Gender-Based Discrimination in the Display of STEM Career Ads. Management Science 65 (7), pp. 2966–2981. External Links: Document, ISSN 0025-1909, Link Cited by: §1.
  • M. K. Lee (2018) Understanding perception of algorithmic decisions: Fairness, trust, and emotion in response to algorithmic management. Big Data & Society 5 (1), pp. 2053951718756684. External Links: Document Cited by: §1.
  • T. Li, A. K. Sahu, A. Talwalkar, and V. Smith (2020) Federated Learning: Challenges, Methods, and Future Directions. IEEE Signal Processing Magazine 37 (3), pp. 50–60. External Links: Document, ISSN 1558-0792 Cited by: §2, §3.3.
  • T. Liang, C. Liu, T. Lin, and B. Lin (2007) Effect of team diversity on software project performance. Industrial Management & Data Systems 107 (5), pp. 636–653. External Links: Document, ISSN 0263-5577, Link Cited by: §3.3.
  • Z. C. Lipton (2018) The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery.. Queue 16 (3), pp. 31–57. External Links: Document, ISSN 1542-7730, Link Cited by: §2.
  • J. M. Logg (2017) Theory of Machine: When Do People Rely on Algorithms?. External Links: Link Cited by: §1.
  • J. M. Logg (2016) When do people rely on algorithms?. PhD Thesis, University of California, Berkeley. Cited by: §1.
  • M. Loi and M. Spielkamp (2021) Towards Accountability in the Use of Artificial Intelligence for Public Administrations. In Proc. AAAI/ACM Conference on AI, Ethics, and Society, pp. 757–766. External Links: Document Cited by: §2, §2.
  • L. Longo, R. Goebel, F. Lecue, P. Kieseberg, and A. Holzinger (2020) Explainable Artificial Intelligence: Concepts, Applications, Research Challenges and Visions. In Proc. Cross Domain Conference for Machine Learning and Knowledge Extraction, pp. 1–16 (english). External Links: Document, ISBN 978-3-030-57321-8 Cited by: §2.
  • S. M. Lundberg and S. Lee (2017) A unified approach to interpreting model predictions. In Proc. International Conference on Neural Information Processing Systems, pp. 4768–4777. External Links: ISBN 978-1-5108-6096-4, Link Cited by: §3.3.
  • M. M. Maas (2021) Aligning AI Regulation to Sociotechnical Change. Social Science Research Network (english). External Links: Link Cited by: §2.
  • M. MacCarthy (2019) An Examination of the Algorithmic Accountability Act of 2019. Social Science Research Network (english). External Links: Link Cited by: §2.
  • R. Marcinkevičs and J. E. Vogt (2020) Interpretability and Explainability: A Machine Learning Zoo Mini-tour. External Links: 2012.01805 Cited by: §2.
  • J. Mattis (2018) Summary of the 2018 National Defense Strategy of the United States of America. Department of Defense Washington United States (english). External Links: Link Cited by: §1.
  • L. McGregor, D. Murray, and V. Ng (2019) International Human Rights Law as a Framework for Algorithmic Accountability. International & Comparative Law Quarterly 68 (2), pp. 309–343 (english). External Links: Document, ISSN 0020-5893, 1471-6895, Link Cited by: §2, §2, §3.3.
  • N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, and A. Galstyan (2021) A Survey on Bias and Fairness in Machine Learning. ACM Computing Surveys 54 (6), pp. 115:1–115:35. External Links: Document, ISSN 0360-0300, Link Cited by: §2, §2, §3.2, §3.3.
  • M. Mitchell, S. Wu, A. Zaldivar, P. Barnes, L. Vasserman, B. Hutchinson, E. Spitzer, I. D. Raji, and T. Gebru (2019) Model Cards for Model Reporting. In Proc. ACM Conference on Fairness, Accountability, and Transparency, pp. 220–229. External Links: Document, ISBN 978-1-4503-6125-5, Link Cited by: §2, §3.1.
  • Modeling and S. Enterprise (2011) Acceptability Criteria. External Links: Link Cited by: §3.4.
  • J. Mökander, J. Morley, M. Taddeo, and L. Floridi (2021) Ethics-Based Auditing of Automated Decision-Making Systems: Nature, Scope, and Limitations. Science and Engineering Ethics 27 (4), pp. 44 (english). External Links: Document, ISSN 1471-5546, Link Cited by: §2, §2.
  • E. Moss, E. A. Watkins, R. Singh, M. C. Elish, and J. Metcalf (2021) Assembling Accountability: Algorithmic Impact Assessment for the Public Interest. Social Science Research Network (english). External Links: Link Cited by: §2, §2.
  • I. Naja, M. Markovic, P. Edwards, and C. Cottrill (2021) A Semantic Framework to Support AI System Accountability and Audit. In Proc. European Semantic Web Conference, pp. 160–176 (english). External Links: Document, ISBN 978-3-030-77385-4 Cited by: §2, §2.
  • C. O’Neil (2016) Weapons of math destruction how big data increases inequality and threatens democracy. Crown Publishing, New York, NY (english). Cited by: §1, §1.
  • I. C. Office (2020) Guidance on the AI auditing framework: Draft guidance for consultation. Information Commissioner’s Office (english). External Links: Link Cited by: §2, §2, §2, §2, §2.
  • I. C. Office (2021) Data protection impact assessments. (english). External Links: Link Cited by: §3.4.
  • H. Quay-de la Vallee and N. Duarte (2019) Algorithmic Systems in Education: Incorporating Equity and Fairness When Using Student Data. Center for Democracy & Technology (american). External Links: Link Cited by: §1.
  • I. D. Raji, A. Smart, R. N. White, M. Mitchell, T. Gebru, B. Hutchinson, J. Smith-Loud, D. Theron, and P. Barnes (2020) Closing the AI accountability gap: defining an end-to-end framework for internal algorithmic auditing. In Proc. ACM Conference on Fairness, Accountability, and Transparency, pp. 33–44. External Links: Document, ISBN 978-1-4503-6936-7, Link Cited by: §2, §2, §2, §2.
  • M. T. Ribeiro, S. Singh, and C. Guestrin (2016) ”Why Should I Trust You?”: Explaining the Predictions of Any Classifier. In Proc. ACM International Conference on Knowledge Discovery and Data Mining, pp. 1135–1144. External Links: Document, ISBN 978-1-4503-4232-2, Link Cited by: §3.3.
  • M. L. Rustad and T. H. Koenig (2019) Towards a Global Data Privacy Standard. Florida Law Review 71 (2), pp. 365–454 (english). External Links: Link Cited by: §2.
  • P. Saleiro, B. Kuester, L. Hinkson, J. London, A. Stevens, A. Anisfeld, K. T. Rodolfa, and R. Ghani (2019) Aequitas: A Bias and Fairness Audit Toolkit. External Links: 1811.05577 Cited by: §2.
  • A. D. Selbst and J. Powles (2017) Meaningful information and the right to explanation. International Data Privacy Law 7 (4), pp. 233–242. External Links: Document, ISSN 2044-3994, Link Cited by: §2.
  • A. D. Selbst and S. Barocas (2018) The Intuitive Appeal of Explainable Machines. External Links: Link Cited by: §2.
  • A. D. Selbst, D. Boyd, S. A. Friedler, S. Venkatasubramanian, and J. Vertesi (2019) Fairness and Abstraction in Sociotechnical Systems. In Proc. ACM Conference on Fairness, Accountability, and Transparency, pp. 59–68. External Links: Document, ISBN 978-1-4503-6125-5, Link Cited by: §2, §2.
  • S. Sharma, J. Henderson, and J. Ghosh (2020) CERTIFAI: A Common Framework to Provide Explanations and Analyse the Fairness and Robustness of Black-box Models. In Proc. AAAI/ACM Conference on AI, Ethics, and Society, pp. 166–172. External Links: ISBN 978-1-4503-7110-0, Link Cited by: §2.
  • H. Shen, A. DeVos, M. Eslami, and K. Holstein (2021) Everyday algorithm auditing: understanding the power of everyday users in surfacing harmful algorithmic behaviors. Proceedings of the ACM on Human-Computer Interaction 5 (CSCW2). External Links: Document, Link Cited by: §2, §2.
  • S. C. Slota, K. R. Fleischmann, S. Greenberg, N. Verma, B. Cummings, L. Li, and C. Shenefiel (2021) Many hands make many fingers to point: challenges in creating accountable AI. AI & Society (english). External Links: Document, ISSN 1435-5655, Link Cited by: §2, §2, §2.
  • [94] O. S. o. State State agency general records retention schedules. External Links: Link Cited by: §3.4.
  • A. Strohmayer, J. Slupska, R. Bellini, L. Coventry, T. Hairston, and A. Dodge (2021) Trust and abusability toolkit: Centering safety in human-data interactions. External Links: Link Cited by: §2.
  • Supreme Audit Institutions of Finland, Germany, the Netherlands, Norway and the UK (2021) Auditing machine learning algorithms. External Links: Link Cited by: §2.
  • E. Tagiou, Y. Kanellopoulos, C. Aridas, and C. Makris (2019) A tool supported framework for the assessment of algorithmic accountability. In Proc. International Conference on Information, Intelligence, Systems and Applications, pp. 1–9. External Links: Document Cited by: §2, §2.
  • J. D. Teachman (1980) Analysis of Population Diversity: Measures of Qualitative Variation. Sociological Methods & Research 8 (3), pp. 341–362 (english). External Links: Document, ISSN 0049-1241, Link Cited by: §3.3.
  • P. P. Uhrowczik (1973) Data Dictionary/Directories. IBM Systems Journal 12 (4), pp. 332–350. External Links: Document, ISSN 0018-8670 Cited by: §3.1.
  • E. P. a. C. o. t. E. Union (2016) Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation). (english). External Links: Link Cited by: §3.2, §3.4.
  • United Nations (1948) Universal Declaration of Human Rights. External Links: Link Cited by: §3.2.
  • S. Wachter, B. Mittelstadt, and C. Russell (2021) Why fairness cannot be automated: Bridging the gap between EU non-discrimination law and AI. Computer Law & Security Review 41, pp. 105567 (english). External Links: Document, ISSN 0267-3649, Link Cited by: §2.
  • P. D. Waggoner, R. Kennedy, H. Le, and M. Shiran (2019) Big Data and Trust in Public Policy Automation. Statistics, Politics and Policy 10 (2), pp. 115–136 (en). External Links: Document, ISSN 2151-7509, Link Cited by: §3.1.
  • D. R. Wallace and J. C. Cherniavsky (1990) Guide to software acceptance. National Institute of Standards and Technology (english). External Links: Link Cited by: §3.4.
  • C. Wang, X. Li, H. Han, S. Wang, L. Wang, C. C. Cao, and L. Chen (2021) Counterfactual Explanations in Explainable AI: A Tutorial. In Proceedings ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 4080–4081. External Links: ISBN 978-1-4503-8332-5, Link Cited by: §3.4.
  • X. Wang, J. Li, X. Kuang, Y. Tan, and J. Li (2019) The security of machine learning in an adversarial setting: A survey. Journal of Parallel and Distributed Computing 130, pp. 12–23 (english). External Links: Document, ISSN 07437315, Link Cited by: §2, §3.3.
  • Y. Wang and K. Chaudhuri (2018) Data poisoning attacks against online learning. External Links: 1808.08994 Cited by: §3.2.
  • D. J. Weitzner, H. Abelson, T. Berners-Lee, J. Feigenbaum, J. Hendler, and G. J. Sussman (2008) Information accountability. Communications of the ACM 51 (6), pp. 82–87 (english). External Links: Document, ISSN 0001-0782, 1557-7317, Link Cited by: §2, §2.
  • R. Wexler (2018) Life, Liberty, and Trade Secrets: Intellectual Property in the Criminal Justice System. Stanford Law Review 70 (5), pp. 1343–1430 (english). External Links: Link Cited by: §1.
  • M. Wieringa (2020) What to account for when accounting for algorithms: a systematic literature review on algorithmic accountability. In Proc. ACM Conference on Fairness, Accountability, and Transparency, pp. 1–18. External Links: Document, ISBN 978-1-4503-6936-7, Link Cited by: §2, §3.3.
  • C. Wilson, A. Ghosh, S. Jiang, A. Mislove, L. Baker, J. Szary, K. Trindel, and F. Polli (2021) Building and Auditing Fair Algorithms: A Case Study in Candidate Screening. In Proc. ACM Conference on Fairness, Accountability, and Transparency, pp. 666–677. External Links: Document, ISBN 978-1-4503-8309-7, Link Cited by: §2.
  • A. Winston and I. Burrington (2018) A pioneer in predictive policing is starting a troubling new project. External Links: Link Cited by: §1.
  • A. Xiang and I. D. Raji (2019) On the Legal Compatibility of Fairness Definitions. External Links: 1912.00761 Cited by: §2.
  • H. Xu and S. Mannor (2012) Robustness and generalization. Machine Learning 86 (3), pp. 391–423 (english). External Links: Document, ISSN 1573-0565, Link Cited by: §2.
  • R. V. Zicari, J. Brodersen, J. Brusseau, B. Düdder, T. Eichhorn, T. Ivanov, G. Kararigas, P. Kringen, M. McCullough, F. Möslein, N. Mushtaq, G. Roig, N. Stürtz, K. Tolle, J. J. Tithi, I. van Halem, and M. Westerlund (2021) Z-Inspection®: A Process to Assess Trustworthy AI. IEEE Transactions on Technology and Society 2 (2), pp. 83–97. External Links: Document, ISSN 2637-6415 Cited by: §2, §2.