Increasingly, Artificial Intelligence (AI) is used in order to derive actionable outcomes from data (e.g. categorizations, predictions, decisions). The overall goal of this chapter is to bridge the gap between expert users and lay users, highlighting the explanation needs of both sides and analyzing the current state of explainability. We do this by taking a more detailed look at each component mentioned above and in Figure1. Finally we address some concerns in the context of DNNs.
1.1 The components of explainability
Issues regarding explainable AI (XAI) involve (at least) four components: users, laws and regulations, explanations and algorithms. Together these components provide a context in which explanation methods can be evaluated regarding their adequacy. We briefly discuss these components in Figure 1.
1.2 Users and laws
AI has a serious impact on society, due to the large scale adoption of digital automation techniques that involve information processing and prediction. Deep Neural Networks (DNNs) belong to this set of automation techniques and are used increasingly because of their capability to extract meaningful patterns from raw input. DNNs are fed large quantities of digital information that are easily collected from users. Currently there is much debate regarding the safety of and trust in data processes in general, leading to investigations regarding the explainability of AI-supported decision making. The level of concern about these topics is reflected by official regulations such as the General Data Protection Regulation (GDPR), also mentioned in (Doshi-Velez and Kim, 2017; Holzinger et al., 2017a), incentives to promote the field of explainability (Gunning, 2017)
and institutional initiatives to ensure the safe development of AI such as OpenAI. As the technology becomes more widespread, DNNs in particular, the dependency on said technology increases and ensuring trust in DNN technology becomes a necessity. Current DNNs are achieving unparalleled performance in areas of Computer Vision (CV) and Natural Language Processing (NLP). They are also being used in real-world applications in e.g. medical imaging(Lee et al., 2017), autonomous driving (Bojarski et al., 2017) and legislation (Lockett et al., ).
1.3 Explanation and DNNs
The challenge with DNNs in particular lies in providing insight into the processes leading to their outcomes, and thereby helping to clarify under which circumstances they can be trusted to perform as intended and when they cannot. Unlike other methods in Machine Learning (ML), such as decision trees or Bayesian networks, an explanation for a certain decision made by a DNN cannot be retrieved by simply scrutinizing the inference process. The learned internal representations and the flow of information through the network are hard to analyze: As architectures get deeper, the number of learnable parameters increases. It is not uncommon to have networks with millions of parameters. Furthermore, network architecture is determined by various components (unit type, activation function, connectivity pattern, gating mechanisms) and the result of a complex learning procedure, which itself depends on various properties (regularization, adaptive mechanisms, employed cost function). The net result of the interaction between these components cannot be predicted in advance. Because of these complications, DNNs are often calledblack box models, as opposed to glass-box models (Holzinger et al., 2017b). Fortunately, these problems have not escaped the attention of the ML/Deep Learning (DL) community (Zeng, 2016; Samek et al., 2017; Seifert et al., 2017; Olah et al., 2017; Hall et al., 2017; Montavon et al., 2018; Marcus, 2018; Doshi-Velez and Kim, 2017). Research on how to interpret and explain the decision process of Artificial Neural Networks (ANNs) has been going on since the late 1980’s (Elman, 1989; Andrews et al., 1995). The objective of explanation methods is to make specific aspects of a DNN’s internal representations and information flow interpretable by humans.
2 Users and their concerns
Various kinds of DNN users can be distinguished. Users entertain certain values; these include ethical values such as fairness, neutrality, lawfulness, autonomy, privacy or safety, or functional values such as accuracy, usability, speed or predictability. Out of these values certain concerns regarding DNNs may arise, e.g. apprehensions about discrimination or accuracy. These concerns get translated into questions about the system, e.g. “did the factor ‘race’ influence the outcome of the system” or “how reliable was the data used?” In this section we identify at least two general types of users: the expert users and the lay users, that can be further categorized into six specific kinds of users. Note that there could be (and there regularly is) overlap between the users described below, such that a particular user can be classified as belonging to more than one of the categories.
Expert users are the system builders and/or modifiers that have direct influence on the implementation of the network. Two kinds of experts can be identified:
are generally researchers involved in extending the field and have detailed knowledge about the mathematical theories and principles of DNNs. DNN engineers are interested in explanations of a functional nature, e.g. the effects of various hyperparameters on the performance of the network or methods that can be used for model debugging.
DNN developers are generally application builders who make software solutions that can be used by lay people. DNN developers often make use of off-the-shelf DNNs, often re-training the DNN along with tuning certain hyperparameters and integrating them with various software components, resulting in a functional application. The DNN developer is concerned with the goals of the overall application and assesses whether they have been met by the DNN solution. DNN developers are interested in explanation methods that allow them to understand the behavior of the DNN in the various use cases of the integrated software application.
Lay users do not and need not have knowledge of how the DNN was implemented and the underlying mathematical principles, nor do they require knowledge of how the DNN was integrated with other software components resulting in a final functional application. At least four lay users are identified:
The owner of the software application in which the DNN is embedded. The owner is usually an entity that acquires the application for possible commercial, practical or personal use. For example, an owner can be an organization (e.g. a hospital or a car manufacturer) that purchases the application for end users (e.g. employees (doctors) or clients (car buyers)), but the owner can also be a consumer that purchases the application for personal use. In the latter case the categorization of owner fully overlaps with the next category of users which are the end users. The owner is concerned with explainability questions about the capabilities of the application, e.g. justification of a prediction or a prediction given the input data, and aspects of accountability, e.g. to what extent can application malfunction be attributed to the DNN component?
The end user for whom the application was intended to be used by. The end user uses the application as part of their profession or for personal use. The end user is concerned with explainability about the capabilities of the application, e.g. justification of a prediction given the input data, and explainability regarding the behavior of the application, e.g. why does the application not do what it was advertised to do?
The data subject is the entity whose information is being processed by the application or the entity which is directly affected by the application outcome. An outcome is the output of the application in the context of the use case. Sometimes the data subject is the same entity as the end user, for example in the case that the application is meant for personal use. The data subject is mostly concerned with the ethical and moral aspects that result from the actionable outcomes. An actionable outcome is an outcome that has consequences or an outcome on which important decisions are based.
Stakeholders are people or organizations without a direct connection to either the development, use or outcome of the application and who can reasonably claim an interest in the process, for instance when its use runs counter to particular values they protect. Governmental and non-governmental organizations may put forward legitimate information requests regarding the operations and consequences of DNNs. Stakeholders are often interested in the ethical and legal concerns raised in any phase of the process.
Case study: autonomous driving
In this section the different users are presented in the context of a self-driving car.
The DNN engineer creates a DL solution to the problem of object segmentation and object classification by experimenting with various types of networks. Given raw video input the DL solution gives the output of the type of object and the location of the object in the video.
The DNN developer creates a planning system which integrates the output of the DL solution with other components in the system. The planning system decides which actions the car will take.
The owner acquires the planning system and produces a car in which the planning system is operational.
The end user purchases the car and uses the car to travel from point A to point B.
The data subjects are all the entities from which information is captured along the route from point A to point B: pedestrians, private property such as houses, other cars.
The stakeholders are governmental institutions which formulate laws regulating the use of autonomous vehicles, or insurance companies that have to assess risk levels and their consequences.
3 Laws and regulations
An important initiative within the European Union is the General Data Protection Regulation (GDPR) 222https://www.eugdpr.org that was approved on April 14, 2016, and became enforceable on May 25, 2018. The GDPR distinguishes between personal data, data subjects, data processors and data controllers (Article 4, Definitions, Paragraphs 1, 7 & 8). Personal data is defined as “any information relating to an identified or identifiable natural person (data subject)”. A data processor is the natural or legal person, public authority, agency or other body which processes data on behalf of the data controller, who determines the purposes, conditions and means of the processing. Hence, the DNN can function as a tool to be used by the data processor, whereas owners or end users can fill the role of data controllers.
The GDPR focuses in part on profiling: “any form of automated processing of personal data consisting of the use of personal data to evaluate certain personal aspects relating to a natural person, in particular to analyse or predict aspects concerning that natural person’s performance at work, economic situation, health, personal preferences, interests, reliability, behaviour, location or movements” (Article 4, Definitions, Paragraph 4). According to articles 13, 14 and 15, when personal data is collected from a data subject for automated decision-making, the data subject has the right to access, and the data controller is obliged to provide, “meaningful information about the logic involved.” Article 12 stipulates that the provision of information to data subjects should be in “concise, transparent, intelligible and easily accessible form, using clear and plain language.”
The right to meaningful information translates into the demand that actionable outcomes of DNNs need to be explained, i.e. be made transparent, interpretable or comprehensible to humans. Transparency refers to the extent to which an explanation makes a specific outcome understandable to a particular (group of) users. Understanding, in this context, amounts to a person grasping how a particular outcome was reached by the DNN. Note that this need not imply agreeing with the conclusion, i.e. accepting the outcome as valid or justified. In general, transparency may be considered as recommendable, leading to e.g. a greater (societal) sense of control and acceptance of ML applications. Transparency is normally also a precondition for accountability: i.e. the extent to which the responsibility for the actionable outcome can be attributed to legally (or morally) relevant agents (governments, companies, expert or lay users, etc.). However, transparency may also have negative consequences, e.g. regarding privacy or by creating possibilities for manipulation (of data, processing or training).
In relation to the (perceived) need for explanation, two reasons for investigation stand out in particular. First, a DNN may appear to dysfunction, i.e. fail to operate as intended, e.g. through bugs in the code (process malfunction). Second, it may misfunction, e.g. by producing unintended or undesired (side-)effects (Floridi et al., 2015; Mittelstadt et al., 2016) that are deemed to be societally or ethically unacceptable (outcome malfunction). Related to dysfunction is a first category of explanations. This category is based on the information necessary in order to understand the system’s basic processes, e.g. to assess whether it is functioning properly, as intended, or whether it dysfunctions (e.g. suboptimal or erroneous results). This type of explanation is normally required by DNN developers and expert users. The information is used to interpret, predict, monitor, diagnose, improve, debug or repair the functioning of a system (Weller, 2017).
Once an application is made available to non-expert users, normally certain guarantees regarding the system’s proper functioning are in place. Generally speaking, owners, end users, data subjects and stakeholders are more interested in a second category of explanations, where suspicions about a DNN’s misfunctioning (undesired outcomes) leads to requests for “local explanations”. Users may request information about how a particular outcome was reached by the DNN, which aspects of input data, which learning factors or other parameters of the system influenced its decision or prediction. This information is then used to assess the appropriateness of the outcome in relation to the concerns and values of users (Doran et al., 2017; Wachter et al., 2017; Doshi-Velez et al., 2017; Weller, 2017). The aim of local explanations is to strengthen the confidence and trust of users that the system is not (or will not be) conflicting with their values, i.e. that it does not violate fairness or neutrality. Note that this implies that the offered explanations should match (within certain limits) the particular user’s capacity for understanding (Doshi-Velez and Kim, 2017), as indicated by the GDPR.
5 Explanation methods
So far the users, the GDPR, and the role of explanations have been discussed. To bridge the gap from that area to the more technical area of explanation methods, we need to be able to evaluate the capabilities of existing methods, in the context of the users and their needs. We bridge the gap in two ways. First, we identify, on a high level, desirable properties of explanation methods. Second, we introduce a taxonomy to categorize all types of explanation methods and third, assess the presence of the desirable properties in the categories in our taxonomy.
5.1 Desirable properties of explainers
Based on a survey of the literature, we arrive at the following properties which any explainer should have:
High Fidelity The degree to which the interpretation method agrees with the input-output mapping of the DNN. This term appears in (Arbatli and Akin, 1997; Markowska-Kaczmar and Wnuk-Lipiński, 2004; Zilke et al., 2016; Ribeiro et al., 2016a, b; Andrews et al., 1995; Lakkaraju et al., 2017). Fidelity is arguably the most important property that an explanation model should possess. If an explanation method is not faithful to the original model then it cannot give valid explanations because the input-output mapping is incorrect.
High Interpretabiliy To what extent a user is able to obtain true insight into how actionable outcomes are obtained. We distinguish interpretability into the following two subproperties:
High Clarity The degree to which the resulting explanation is unambiguous. This property is extremely important in safety-critical applications (Andrews et al., 1995) where ambiguity is to be avoided. (Lakkaraju et al., 2017) introduces a quantifiable measure of clarity (unambiguity) for their method.
High Parsimony This refers to the complexity of the resulting explanation. An explanation that is parsimonious is a simple explanation. This concept is generally related to Occam’s razor and in the case of explaining DNNs the principle is also of importance. The optimal degree of parsimony can in part be dependent on the user’s capabilities.
High Generalizability The range of architectures to which the explanation method can be applied. This increases the usefulness of the explanation method. Methods that are model-agnostic (Ribeiro et al., 2016b) are the highest in generalizability.
High Explanatory Power In this context this means how many phenomena the method can explain. This roughly translates to how many different kinds of questions the method can answer. Previously in Section 2 we have identified a number of questions that users may have. It is also linked to the notion that the explainer should be able to take a global perspective (Ribeiro et al., 2016b), in the sense that it can explain the behaviour of the model rather than only accounting for individual predictions.
5.2 A taxonomy for explanation methods
Over a relatively short period of time a plethora of explanation methods and strategies have come into existence, driven by the need of expert users to analyze and debug their DNNs. However, apart from a non-exhaustive overview of existing methods (Montavon et al., 2018) and classification schemes for purely visual methods (Grün et al., 2016; Seifert et al., 2017; Zeng, 2016; Kindermans et al., 2017), little is known about efforts to rigorously map the landscape of explanation methods and isolate the underlying patterns that guide explanation methods. In this section a taxonomy for explanation methods is proposed. Three main classes of explanation methods are identified and their features described. The taxonomy was derived by analyzing the historical and contemporary trends surrounding the topic of interpretation of DNNs and explainable AI. We realize that we cannot foresee the future developments of DNNs and their explainability methods. As such it is possible that in the future the taxonomy needs to be modified. We propose the following taxonomy:
- Rule-extraction methods
Extract rules that approximate the decision-making process in a DNN by utilizing the input and output of the DNN.
- Attribution methods
Measures the importance of a component by changing to the input or internal components and recording how much the changes affect model performance. Methods known by other names that fall in this category are occlusion, perturbation, erasure, ablation and influence. Attribution methods are often visualized and sometimes referred to as visualization methods.
- Intrinsic methods
Aim to improve the interpretability of internal representations with methods that are part of the DNN architecture. Intrinsic methods increase fidelity, clarity and parsimony in attribution methods.
In the following subsections we will describe the main features of each class and give examples from current research.
5.2.1 Rule-extraction methods
Rule-extraction methods extract human interpretable rules that approximate the decision-making process in a DNN. Older genetic algorithm based rule extraction methods for ANNs can be found in(Andrews et al., 1995; Arbatli and Akin, 1997; Lu et al., 2006). Andrews et al. (1995) specify three categories of rule extraction methods:
- Decompositional approach
Decomposition refers to breaking down the network into smaller individual parts. For the decompositional approach, the architecture of the network and/or its outputs are used in the process. Zilke et al. (2016) uses a decompositional algorithm that extracts rules for each layer in the DNN. These rules are merged together in a final merging step to produce a set of rules that describe the network behaviour by means of its inputs. Murdoch and Szlam (2017) succeeded in extracting rules from an LSTM by applying a decompositional approach.
- Pedagogical approach
Introduced by Craven and Shavlik (1994) and named by Andrews et al. (1995) the pedagogical approach involves “viewing rule extraction as a learning task where the target concept is the function computed by the network and the input features are simply the network’s input features” (Craven and Shavlik, 1994). The pedagogical approach has the advantage that it is inherently model-agnostic. Recent examples are found in (Ribeiro et al., 2016a; Lakkaraju et al., 2017).
- Eclectic approach
In terms of fidelity, local explanations are more faithful than global explanations. For rule-extraction this means that rules that govern the result of a specific input, or a neighborhood of inputs are more faithful than rules that govern all possible inputs. Rule extraction is arguably the most interpretable category of methods in our taxonomy considering that the resulting set of rules can be unambiguously be interpreted by a human being as a kind of formal language. Therefore we can say that it has a high degree of clarity. In terms of parsimony we can say that if the ruleset is ”small enough” the parsimony is higher than when the ruleset is “too large”. What determines “small enough” and “too large” is difficult to quantify formally and is also dependent on the user (expert vs. lay). In terms of generalizability it can go both ways: if a decompositional approach is used it is likely that the method is not generalizable, while if a pedagogical approach is used the method is highly generalizable. In terms of explanatory power, rule-extraction methods can 1) validate whether the network is working as expected in terms of overall logic flow, and 2) explain which aspects of the input data had an effect that lead to the specific output.
5.2.2 Attribution methods
Attribution, a term introduced by Ancona et al. (2018), also referred to as relevance (Bach et al., 2015; Binder et al., 2016; Zintgraf et al., 2017; Robnik-Šikonja and Kononenko, 2008), contribution (Shrikumar et al., 2017), class saliency (Simonyan et al., 2013) or influence (Kindermans et al., 2016; Adler et al., 2016; Koh and Liang, 2017), aims to reveal components of high importance in the input to the DNN and their effect as the input is propagated through the network. Because of this property we can categorize the following methods to the attribution category: occlusion (Guçlütürk et al., 2017), erasure (Li et al., 2016), perturbation (Fong and Vedaldi, 2017), adversarial examples (Papernot et al., 2017) and prediction difference analysis (Zintgraf et al., 2017). Other methods that belong to this category are found in (Baehrens et al., 2010; Murdoch et al., 2018; Ribeiro et al., 2016b). It is worth mentioning that attribution methods do not only apply to image input but also to other forms of input, such as text processing by LSTMs (Murdoch et al., 2018). The definition of attribution methods in this chapter is similar to that of saliency methods (Kindermans et al., 2017), but more general than the definition of attribution methods in (Kindermans et al., 2017) akin to the definition in (Ancona et al., 2018).
The majority of explanation methods for DNNs visualize the information obtained by attribution methods. Visualization methods were popularized by (Erhan et al., 2009; Simonyan et al., 2013; Zeiler and Fergus, 2014) in recent years and are concerned with how the important features are visualized. Zeng (2016) identifies that current methods focus on three aspects of visualization: feature visualization, relationship visualization and process visualization. Overall visualization methods are very intuitive methods to gain a variety of insight about a DNN decision process on many levels including architecture assessment, model quality assessment and even user feedback integration, e.g. Olah et al. (2018) create intuitive visualization interfaces for image processing DNNs.
Kindermans et al. (2017) has shown recently that attribution methods “lack reliability when the explanation is sensitive to factors that do not contribute to the model prediction.” Furthermore they introduce the notion of input invariance as a prerequisite for accurate attribution. In other words, if the attribution method does not satisfy input invariance, we can consider it to have low fidelity. In terms of clarity, there is a degree of ambiguity that is inherent with these methods because visual explanations can be interpreted in multiple ways by different users, even by users in the same user category. In contrast to the precise results of rule-extraction methods, the information that results from attribution methods has less structure. In addition, the degree of clarity is dependent on the degree of fidelity of the method: low fidelity can cause incorrect attribution, resulting in noisy output with distracting attributions that increase ambiguity. The degree of parsimony depends on the method of visualization itself. Methods that visualize only the significant attributions exhibit a higher degree of parsimony. The degree of generalizability depends on which components are used to determine attribution. Methods that only use the input and output are inherently model agnostic, resulting in the highest degree of generalizability. Following this logic, methods that make use of internal components are generalizable to the degree that other models share these components. For example, deconvolutional networks (Zeiler et al., 2010) can be applied to models that make use of convolutions to extract features from input images. In terms of explanatory power, this class of methods can reflect intuitively with visual explanations which factors in the input dimension had a significant impact on the output of the DNN. However these methods do not explain the reason for the importance of the particular factor attribution.
5.2.3 Intrinsic methods
The previous categories are designed to make explainable some aspects of a DNN in a process separate from training the DNN. In contrast, this category aims to improve the interpretability of internal representations with methods that are part of the DNN architecture, e.g. as part of the loss function(Dong et al., 2017b, a), modules that add additional capabilities (Santoro et al., 2017; Palm et al., 2017), or as part of the architecture structure, in terms of operations between layers (Li et al., 2017; Wu et al., 2017; Louizos et al., 2017; Goudet et al., 2017). Dong et al. (2017b) provide an interpretive loss function to increase the visual fidelity of the learned features. More importantly Dong et al. (2017a)
show that by training DNNs with adversarial data and a consistent loss, we can trace back errors made by the DNN to individual neurons and identify whether the data was adversarial.Santoro et al. (2017) give a DNN the ability to answer relational reasoning questions about a specific environment, by introducing a relational reasoning module that learns a relational function, which can be applied to any DNN. Palm et al. (2017) build on work by Santoro et al. (2017) and introduces a recurrent relational network which can take the temporal component into account. Li et al. (2017) introduce an explicit structure to DNNs for visual recognition by building in an AND-OR grammar directly in the network structure. This leads to better interpretation of the information flow in the network, hence increased parsimony in attribution methods. Louizos et al. (2017) make use of generative neural networks perform causal inference and Goudet et al. (2017) use generative neural networks to learn functional causal models. Intrinsic methods do not explicitly explain anything by themselves. Instead they increase fidelity, clarity and parsimony in attribution methods. This class of methods is different from attribution methods because it tries to make the DNN inherently more interpretable by changing the architecture of the DNN, where attribution methods use what is there already and only transform aspects of the representation to something meaningful after the network is trained.
6 Addressing general concerns
As indicated in Figure 1, users have certain values, that in relation to a particular technology may lead to concerns, that in relation to particular applications can lead to specific questions. Mittelstadt et al. (2016) and Danks and London (2017) distinguish various concerns that users may have. The kinds of concerns they discuss focus to a large extent on the inconclusiveness, inscrutability or misguidedness of used evidence. That is, they concern to a significant extent the reliability and accessibility of used data (data mining, generally speaking). In addition to apprehensions about data, there are concerns that involve aspects of the processing itself, e.g. the inferential validity of an algorithm. Also, questions may be raised about the validity of a training process (e.g. requiring information about how exactly a DNN is trained). In the following, we provide a list of general concerns that should be addressed when developing predictive models such as DNNs:
- Flawed data collection
Data collection may be flawed in several ways. Large labeled datasets that are used to train DNNs are either acquired by researchers (often via crowdsourcing) or by companies that ‘own’ the data. However, data quality may depend on multiple factors such as noise or censoring and there is no strict control on whether data is annotated correctly. Furthermore, the characteristics of the workers who annotated the data may introduce unwanted biases (Barocas and Selbst, 2016). These biases may be due to preferences that do not generalize across cultures or due to stereotyping, where sensitivity to irrelevant attributes such as race or gender may induce unfair actionable outcomes. The same holds for the selection of the data that is used for annotation in the first place. Used data may reflect the status quo, which is not necessarily devoid of biases (Caliskan et al., 2017). Furthermore, selection bias may have as a result that data collected in one setting need not generalize to other settings. For example, video data used to train autonomous driving systems may not generalize to other locations or conditions.
- Inscrutable data use
The exact use of the data to train DNNs may also be opaque. Users may worry about what (part of the) data exactly has led to the outcome. Often it is not even known to the data subject which personal data is being used for what purposes. A case in point is the use of person data for risk profiling by governmental institutions. For example, criticisms have been raised about the way the Dutch SyRI system uses data to detect fraud.333https://bijvoorbaatverdacht.nl Furthermore, the involvement of expert users who may be prone to biases as well may have an implicit influence on DNN training.
- Suboptimal inferences
The inferences made by DNNs are of a correlational rather than a causal nature. This implies that subtle correlations between input features may influence network output, which themselves may be driven by various biases. Work is in progress to mitigate or remove the influence of sensitive variables that should not affect decision outcomes by embracing causal inference procedures (Chiappa and Gillam, 2018). Note further that the impact of suboptimal inferences is domain dependent. For example, in medicine and the social sciences, suboptimal inferences may directly affect the lives of individuals or whole populations whereas in the exact sciences, suboptimal inferences may affect evidence for or against a specific scientific theory.
- Undesirable outcomes
End users or data subjects may feel that the outcome of the DNN is somehow undesirable in relation to the particular values they hold, e.g. violating fairness or privacy. Importantly, actionable outcomes should take into account preferences of the stakeholder, which can be an individual (e.g. when deciding on further medical investigation) as well as the community as a whole (e.g. in case of policies about autonomous driving or predictive policing). These considerations demand the involvement of domain experts and ethicists already in the earliest stages of model development. Finally, model predictions may be of a statistical rather than deterministic nature. This speaks for the inclusion of decision-theoretic constructs in deciding on optimal actionable outcomes (von Neumann and Morgenstern, 1953).
- Adversarial attacks
Images (Szegedy et al., 2013; Cubuk et al., 2017) and audio (Carlini and Wagner, 2018) can easily be distorted with modifications that are imperceptible to humans. Such distortions cause DNNs to make incorrect inferences and can be done with the purpose of intentionally misleading DNNs (e.g. yielding predictions in favor of the perpetrator). Work in progress shows that there are methods to detect adversarial instances (Rawat et al., 2017) and to mitigate the attacks (Lin et al., 2017). However further research is needed to increase the robustness of DNNs against adversarial attacks as there are no methods in existence that fully diminish the effects of adversarial attacks.
As stated by Doran et al. (2017), explanation methods may make predictive models such as DNNs more comprehensible. However, explanation methods alone not completely resolve the raised concerns.
In this chapter we set out to analyze the question of “What can be explained?” given the users and their needs, laws and regulations, and existing explanation methods. Specifically, we looked at the capabilities of explanation methods and analyzed which questions/concerns about explainability these methods address in the context of DNNs.
Overall, it is clear that (visual) explanations can be given about various aspects of the influence of the input on the output (e.g. given the input data, which aspects of the data lead to the output?), by making use of both rule-extraction and attribution methods. Also, when used in combination with attribution methods, intrinsic methods lead to more explainable DNNs. It is likely that in the future we will see the rise of a new category of explanation methods that combine aspects of rule-extraction, attribution and intrinsic methods, to answer specific questions in a simple human interpretable language.
Furthermore, it is obvious that current explanation methods are tailored to expert users, since the interpretation of the results require knowledge of the DNN process. As far as we are aware, explanation methods, e.g. intuitive explanation interfaces, for lay users do not exist. Ideally, if such explanation methods would exist, they should be able to answer, in a simple human language, questions about every operation that the application performs. This is not an easy task since the number of conceivable questions one could ask about the working of an application is substantial.
Two particular concerns, which are difficult to address with explanation methods, is the concern about bias in datasets that leads to biased DNNs, as well as the suspicion about unfair outcomes: Can we indicate that the DNN is biased, and if so, can we remove the bias? Has the DNN been applied responsibly? These are not problems that are directly solvable with explanation methods. However, explanation methods alleviate the first problem to the extent that learned features can be visualized (using attribution methods) and further analyzed for bias using other methods that are not explanation methods. For the second problem, more general measures, such as regulations and laws, will need to be developed.
- Adler et al. (2016) Philip Adler, Casey Falk, Sorelle A. Friedler, Gabriel Rybeck, Carlos Scheidegger, Brandon Smith, and Suresh Venkatasubramanian. Auditing black-box models for indirect influence. In 2016 IEEE 16th International Conference on Data Mining (ICDM). IEEE, 2016.
- Ancona et al. (2018) Marco Ancona, Enea Ceolini, Cengiz Oztireli, and Markus Gross. Towards better understanding of gradient-based attribution methods for deep neural networks. In 6th International Conference on Learning Representations (ICLR 2018), 2018.
- Andrews et al. (1995) Robert Andrews, Joachim Diederich, and Alan B. Tickle. Survey and critique of techniques for extracting rules from trained artificial neural networks. Knowledge-Based Systems, 8(6):373–389, 1995.
- Arbatli and Akin (1997) A. Duygu Arbatli and H. Levent Akin. Rule extraction from trained neural networks using genetic algorithms. Nonlinear Analysis: Theory, Methods & Applications, 30(3):1639–1648, 1997.
- Bach et al. (2015) Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE, 10(7), 2015.
- Baehrens et al. (2010) David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller. How to explain individual classification decisions. Journal of Machine Learning Research (JMLR), 11:1803–1831, 2010.
- Barocas and Selbst (2016) Solon Barocas and Andrew D. Selbst. Big data’s disparate impact. Cal. L. Rev., 104:671, 2016.
- Binder et al. (2016) Alexander Binder, Sebastian Bach, Grégoire Montavon, Klaus-Robert Müller, and Wojciech Samek. Layer-wise relevance propagation for deep neural network architectures. In Information Science and Applications (ICISA) 2016, pages 913–922. Springer, 2016.
- Bojarski et al. (2017) Mariusz Bojarski, Philip Yeres, Anna Choromanska, Krzysztof Choromanski, Bernhard Firner, Lawrence Jackel, and Urs Muller. Explaining how a deep neural network trained with end-to-end learning steers a car. arXiv preprint arXiv:1704.07911, 2017.
- Caliskan et al. (2017) Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186, 2017.
- Carlini and Wagner (2018) Nicholas Carlini and David Wagner. Audio adversarial examples: Targeted attacks on speech-to-text. arXiv preprint arXiv:1801.01944, 2018.
- Chiappa and Gillam (2018) Silvia Chiappa and Thomas P.S. Gillam. Path-specific counterfactual fairness. arXiv preprint arXiv:1802.08139, 2018.
- Craven and Shavlik (1994) Mark W. Craven and Jude W. Shavlik. Using sampling and queries to extract rules from trained neural networks. In Machine Learning Proceedings 1994, pages 37–45. Elsevier, 1994.
- Cubuk et al. (2017) Ekin D. Cubuk, Barret Zoph, Samuel S. Schoenholz, and Quoc V. Le. Intriguing properties of adversarial examples. arXiv preprint arXiv:1711.02846, 2017.
- Danks and London (2017) David Danks and Alex John London. Algorithmic bias in autonomous systems. In Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI), pages 4691–4697. AAAI Press, 2017.
- Dong et al. (2017a) Yinpeng Dong, Hang Su, Jun Zhu, and Fan Bao. Towards interpretable deep neural networks by leveraging adversarial examples. arXiv preprint arXiv:1708.05493, 2017a.
Dong et al. (2017b)
Yinpeng Dong, Hang Su, Jun Zhu, and Bo Zhang.
Improving interpretability of deep neural networks with semantic
2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017b.
- Doran et al. (2017) Derek Doran, Sarah Schulz, and Tarek R. Besold. What does explainable AI really mean? a new conceptualization of perspectives. arXiv preprint arXiv:1710.00794, 2017.
- Doshi-Velez and Kim (2017) Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608, 2017.
- Doshi-Velez et al. (2017) Finale Doshi-Velez, Mason Kortz, Ryan Budish, Christopher Bavitz, Samuel J. Gershman, David O’Brien, Stuart Shieber, Jim Waldo, David Weinberger, and Alexandra Wood. Accountability of AI under the law: The role of explanation. SSRN Electronic Journal, 2017.
- Elman (1989) Jeffrey L. Elman. Representation and structure in connectionist models. Technical report, 1989.
- Erhan et al. (2009) Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. University of Montreal, 1341:3, 2009.
- Floridi et al. (2015) Luciano Floridi, Nir Fresco, and Giuseppe Primiero. On malfunctioning software. Synthese, 192(4):1199–1220, 2015.
- Fong and Vedaldi (2017) Ruth C. Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, 2017.
- Goudet et al. (2017) Olivier Goudet, Diviyan Kalainathan, Philippe Caillou, David Lopez-Paz, Isabelle Guyon, Michele Sebag, Aris Tritas, and Paola Tubaro. Learning functional causal models with generative neural networks. arXiv preprint arXiv:1709.05321, 2017.
- Grün et al. (2016) Felix Grün, Christian Rupprecht, Nassir Navab, and Federico Tombari. A taxonomy and library for visualizing learned features in convolutional neural networks. arXiv preprint arXiv:1606.07757, 2016.
- Guçlütürk et al. (2017) Yaǧmur Guçlütürk, Umut Güçlü, Marc Perez, Hugo Jair Escalante, Xavier Baro, Isabelle Guyon, Carlos Andujar, Julio Jacques Junior, Meysam Madadi, Sergio Escalera, Marcel A. J. van Gerven, and Rob van Lier. Visualizing apparent personality analysis with deep residual networks. In 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), pages 3101–3109, 2017.
- Gunning (2017) David Gunning. Explainable artificial intelligence (XAI). Defense Advanced Research Projects Agency (DARPA), 2017.
- Hall et al. (2017) Patrick Hall, Wen Phan, and SriSatish Ambati. Ideas on interpreting machine learning. 2017. Available online at: https://www.oreilly.com/ideas/ideas-on-interpreting-machine-learning.
- Holzinger et al. (2017a) Andreas Holzinger, Chris Biemann, Constantinos S. Pattichis, and Douglas B. Kell. What do we need to build explainable AI systems for the medical domain? arXiv preprint arXiv:1712.09923, 2017a.
- Holzinger et al. (2017b) Andreas Holzinger, Markus Plass, Katharina Holzinger, Gloria Cerasela Crişan, Camelia-Mihaela Pintea, and Vasile Palade. A glass-box interactive machine learning approach for solving NP-hard problems with the human-in-the-loop. arXiv preprint arXiv:1708.01104, 2017b.
- Kindermans et al. (2016) Pieter-Jan Kindermans, Kristof T. Schütt, Klaus-Robert Müller, and Sven Dähne. Investigating the influence of noise and distractors on the interpretation of neural networks. arXiv preprint arXiv:1611.07270, 2016.
- Kindermans et al. (2017) Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T. Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. arXiv preprint arXiv:1711.00867, 2017.
- Koh and Liang (2017) Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning (ICML), volume 70 of Proceedings of Machine Learning Research (PMLR), pages 1885–1894, 2017.
- Lakkaraju et al. (2017) Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Jure Leskovec. Interpretable & explorable approximations of black box models. arXiv preprint arXiv:1707.01154, 2017.
- Lee et al. (2017) Hyunkwang Lee, Shahein Tajmir, Jenny Lee, Maurice Zissen, Bethel Ayele Yeshiwas, Tarik K. Alkasab, Garry Choy, and Synho Do. Fully automated deep learning system for bone age assessment. Journal of Digital Imaging (JDI), 30(4):427–441, 2017.
- Li et al. (2016) Jiwei Li, Will Monroe, and Dan Jurafsky. Understanding neural networks through representation erasure. arXiv preprint arXiv:1612.08220, 2016.
- Li et al. (2017) Xilai Li, Tianfu Wu, Xi Song, and Hamid Krim. AOGNets: Deep AND-OR grammar networks for visual recognition. arXiv preprint arXiv:1711.05847, 2017.
- Lin et al. (2017) Yen-Chen Lin, Ming-Yu Liu, Min Sun, and Jia-Bin Huang. Detecting adversarial attacks on neural network policies with visual foresight. arXiv preprint arXiv:1710.00814, 2017.
- (40) Alan Lockett, Trevor Jefferies, Neil Etheridge, and Alicia Brewer. White paper tag predictions: How DISCO AI is bringing deep learning to legal technology. Available online at: https://www.csdisco.com/disco-ai.
- Louizos et al. (2017) Christos Louizos, Uri Shalit, Joris M. Mooij, David Sontag, Richard Zemel, and Max Welling. Causal effect inference with deep latent-variable models. In Advances in Neural Information Processing Systems 30 (NIPS), pages 6446–6456. 2017.
Lu et al. (2006)
Jianjun Lu, Shozo Tokinaga, and Yoshikazu Ikeda.
Explanatory rule extraction based on the trained neural network and the genetic programming.Journal of the Operations Research Society of Japan (JORSJ), 49(1):66–82, 2006.
- Marcus (2018) Gary Marcus. Deep learning: A critical appraisal. arXiv preprint arXiv:1801.00631, 2018.
- Markowska-Kaczmar and Wnuk-Lipiński (2004) Urszula Markowska-Kaczmar and Paweł Wnuk-Lipiński. Rule extraction from neural network by genetic algorithm with pareto optimization. Artificial Intelligence and Soft Computing-ICAISC 2004, pages 450–455, 2004.
- Mittelstadt et al. (2016) Brent Daniel Mittelstadt, Patrick Allo, Mariarosaria Taddeo, Sandra Wachter, and Luciano Floridi. The ethics of algorithms: Mapping the debate. Big Data & Society, 3(2), 2016.
- Montavon et al. (2018) Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. Methods for interpreting and understanding deep neural networks. Digital Signal Processing, 73:1–15, 2018.
Murdoch and Szlam (2017)
W. James Murdoch and Arthur Szlam.
Automatic rule extraction from long short term memory networks.In International Conference on Learning Representations (ICLR), 2017.
- Murdoch et al. (2018) W. James Murdoch, Peter J. Liu, and Bin Yu. Beyond word importance: Contextual decomposition to extract interactions from LSTMs. In International Conference on Learning Representations (ICLR), 2018.
- Olah et al. (2017) Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. Available online at: https://distill.pub/2017/feature-visualization.
- Olah et al. (2018) Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The building blocks of interpretability. Distill, 2018. Available online at: https://distill.pub/2018/building-blocks.
- Palm et al. (2017) Rasmus Berg Palm, Ulrich Paquet, and Ole Winther. Recurrent relational networks for complex relational reasoning. arXiv preprint arXiv:1711.08028, 2017.
- Papernot et al. (2017) Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security (ASIA CCS ’17), pages 506–519, 2017.
- Rawat et al. (2017) Ambrish Rawat, Martin Wistuba, and Maria-Irina Nicolae. Adversarial phenomenon in the eyes of bayesian deep learning. arXiv preprint arXiv:1711.08244, 2017.
- Ribeiro et al. (2016a) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Nothing else matters: Model-agnostic explanations by identifying prediction invariance. In NIPS Workshop on Interpretable Machine Learning in Complex Systems, 2016a.
- Ribeiro et al. (2016b) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. ”Why should I trust you?” Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’16), pages 1135–1144, 2016b.
- Robnik-Šikonja and Kononenko (2008) Marko Robnik-Šikonja and Igor Kononenko. Explaining classifications for individual instances. IEEE Transactions on Knowledge and Data Engineering, 20(5):589–600, 2008.
- Samek et al. (2017) Wojciech Samek, Thomas Wiegand, and Klaus-Robert Müller. Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models. arXiv preprint arXiv:1708.08296, 2017.
- Santoro et al. (2017) Adam Santoro, David Raposo, David G.T. Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. arXiv preprint arXiv:1706.01427, 2017.
- Seifert et al. (2017) Christin Seifert, Aisha Aamir, Aparna Balagopalan, Dhruv Jain, Abhinav Sharma, Sebastian Grottel, and Stefan Gumhold. Visualizations of deep neural networks in computer vision: A survey. In Transparent Data Mining for Big and Small Data, pages 123–144. Springer, 2017.
- Shrikumar et al. (2017) Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning (ICML), volume 70 of Proceedings of Machine Learning Research (PMLR), 2017.
- Simonyan et al. (2013) Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
- Szegedy et al. (2013) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
- von Neumann and Morgenstern (1953) John von Neumann and Oskar Morgenstern. Theory of Games and Economic Behavior. Princeton University Press, Princeton, NJ, 3rd edition, 1953.
- Wachter et al. (2017) Sandra Wachter, Brent Mittelstadt, and Luciano Floridi. Transparent, explainable, and accountable AI for robotics. Science Robotics, 2(6), 2017.
- Weller (2017) Adrian Weller. Challenges for transparency. Workshop on Human Interpretability in Machine Learning – ICML 2017, 2017.
- Wu et al. (2017) Tianfu Wu, Xilai Li, Xi Song, Wei Sun, Liang Dong, and Bo Li. Interpretable R-CNN. arXiv preprint arXiv:1711.05226, 2017.
- Zeiler and Fergus (2014) Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European Conference on Computer Vision (ECCV), pages 818–833. Springer, 2014.
- Zeiler et al. (2010) Matthew D. Zeiler, Dilip Krishnan, Graham W. Taylor, and Rob Fergus. Deconvolutional networks. In 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2528–2535. IEEE, 2010.
- Zeng (2016) Haipeng Zeng. Towards better understanding of deep learning with visualization. 2016.
- Zilke et al. (2016) Jan Ruben Zilke, Eneldo Loza Mencía, and Frederik Janssen. DeepRED – Rule extraction from deep neural networks. In International Conference on Discovery Science (ICDS), pages 457–473. Springer, 2016.
- Zintgraf et al. (2017) Luisa M. Zintgraf, Taco S. Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. In International Conference on Learning Representations (ICLR), 2017.