Seven challenges for harmonizing explainability requirements

08/11/2021 ∙ by Jiahao Chen, et al. ∙ JPMorgan Chase & Co. J.P. Morgan 5

Regulators have signalled an interest in adopting explainable AI(XAI) techniques to handle the diverse needs for model governance, operational servicing, and compliance in the financial services industry. In this short overview, we review the recent technical literature in XAI and argue that based on our current understanding of the field, the use of XAI techniques in practice necessitate a highly contextualized approach considering the specific needs of stakeholders for particular business applications.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1. Motivation

Artificial intelligence (AI) systems are ubiquitous components of financial, compliance and operational business processes such as marketing (Jannach and Jugovac, 2019), credit decisioning (Hurley and Adebayo, 2016), client satisfaction (Zhao et al., 2019), cybersecurity (Siddiqui et al., 2019), and anti-money laundering (Weber et al., 2019). The growing complexity of these systems pose new challenges for for model governance (Kurshan et al., 2020) fair lending compliance (Chen, 2018), and other regulatory needs.

However, federal regulation in the United States of America (US) has yet to catch up to the frenetic pace of innovation in AI systems, both in research and practical deployments. For example, the OCC 2011-12 / Federal Reserve SR 11-7 Guidance on Model Risk Management (Board of Governers of the Federal Reserve System and Office of the Comptroller of the Currency, 2011)

, published in 2011, is now ten years old and predates the current explosion of deep learning in AI. Similarly, federal fair lending laws like the Equal Credit Opportunity Act (ECOA, a.k.a. Regulation B; enacted in 1974) mandate a customer’s right to explanation when financial institutions take actions that adversely impact their access to credit, and other laws like the Fair Credit Reporting Act (FCRA; enacted in 1970) protect a customer’s right to dispute incorrect data on their credit reports

(Chen, 2018).

The age of financial regulations have hindered regulatory enforcement. A recent investigation from the New York Department of Financial Services into alleged fair lending discrimination in the Apple Card credit card concluded that the letter of fair lending law was not violated insomuch as alleged spousal inequities were fully attributable to different data in the spouses’ credit reports (of Financial Services, 2021), while at the same time noting that “the use of credit scoring in its current form and laws and regulations barring discrimination in lending are in need of strengthening and modernization to improve access to credit” (New York Department of Financial Services, 2021). Recently, federal regulators have issues a joint Request for Information on the use of AI in financial services (The Comptroller of the Currency et al., 2021), signalling an interest to refresh financial regulations in the light of the Biden Administration’s focus on AI (Samp et al., 2021).


In Section 2, we review the recent literature on explainable AI (XAI) and organize the findings into seven major challenges that pose practical considerations when specifying needs for XAI in industry. We conclude in Section 3 that the diversity of core AI technologies, stakeholders and potential applications in the industry, when coupled with the relative immaturity of XAI tools, necessitate an intent-based approach to match stakeholder needs to the most appropriate XAI technique in the context of specific use cases.

2. The challenges

Challenge 1.

The intent and scope of explanations matter.

Different stakeholders have different needs for explanation (Tomsett et al., 2018; Bhatt et al., 2020), but these needs are not often well-articulated or distinguished from each other (Miller, 2019; Ras et al., 2018; Xie et al., 2020; Kaminski and Malgieri, 2020; Kasirzadeh, 2019). Clarity on the intended use of explanation is crucial to select an appropriate XAI tool, as specialized methods exist for specific needs like debugging (Kang et al., 2018), formal verification (safety) (Cousot and Cousot, 1992; Guerra et al., 2009; Xu et al., 2020), uncertainty quantification (Wang and Rockova, 2020; Abdar et al., 2020), actionable recourse (Ustun et al., 2019; Karimi et al., 2020), mechanism inference (Deva et al., 2021), causal inference (Pearl, 2009; Frye et al., 2019; Bertossi et al., 2020), robustness to adversarial inputs (Madry et al., 2018; Lecuyer et al., 2019), data accountability (Yona et al., 2021), social transparency (Ehsan et al., 2021a), interactive personalization (Virgolin et al., 2021), and fairness and algorithmic bias (Ntoutsi et al., 2020) . In contrast, feature importance methods like LIME (Ribeiro et al., 2016) and SHAP (Lundberg and Lee, 2017; Lundberg et al., 2018) focus exclusively on computing quantitative evidence for indicative conditionals (Harper et al., 1981; Bennett, 2003) (of the form “If the applicant doesn’t have enough income, then she won’t get the loan approved”), with some newer counterfactual explanation methods (Artelt and Hammer, 2019; Mothilal et al., 2020; Stepin et al., 2021) and negative contrastive methods (Luss et al., 2019) finding similar evidence for subjunctive conditionals (Pollock, 1976; Byrne, 2016) (of the form “If the applicant increases her income, then she would get the loan approved”) .

Challenge 2.

The type of data and type of model matter.

In addition, the type of explanation changes depending on the data source such as images (Kindermans et al., 2019; Zhu et al., 2021), symbolic logic representation (Albini et al., 2020), or text based explanations (Amir and Amir, 2018)

. Yet other methods exist for domains beyond supervised learning / classification, applied instead to unsupervised learning / clustering

(Horel et al., 2019; Sarhan et al., 2019; Dasgupta et al., 2020)

, reinforcement learning

(Madumal et al., 2020), AI planning (Cashmore et al., 2019)

, computer vision

(Zhu et al., 2021), recommendation systems (Zhang and Chen, 2020)

, natural language processing

(Mullenbach et al., 2018; Yang et al., 2019; Wawrzinek et al., 2020), speech recognition (1), or multi-agent simulations (Amir and Amir, 2018) . Specialized XAI techniques even exist for adaptive systems such as interactive visualization systems (Vig et al., 2021), interactive virtual agents (Kambhampati, 2019; Weitz et al., 2019)

, active learning

(Ghai et al., 2021), and human-in-the-loop systems (Yang et al., 2019) . Furthermore, comparative studies across multiple XAI techniques have shown low mutual agreement (Neely et al., 2021), which is consistent with internal research findings at AI Research. With these research findings taken together, we expect that different explanation tools will be needed to address each problem domain effectively.

Challenge 3.

The human factors around intent and scope of explanations matter.

Results from psychology and other social sciences highlight the social nature of explanation as a critical design criterion (Miller, 2019). Some authors like Kumar et al. (2020) have argued that Shapley value-based explanations do not satisfy human needs for explanation, in particular, the desire to read causality from the explanations. Understanding human-centric factors in explanations for diverse stakeholders is now an area of intense research (Ehsan and Riedl, 2020; Ehsan et al., 2021b) and we expect new relevant results to emerge rapidly. In particular, a recent paper from Microsoft Research (Kaur et al., 2020) provides worrying results that humans generally over-trust AI-generated explanations regardless of their level of subject matter expertise. A detailed human–computer interaction (HCI) study of the use of XAI techniques for explaining fraud models did not show uniform superiority of any one explanation technique (Jesus et al., 2021).

Challenge 4.

There is no consensus around evaluating the correctness of explanations.

One of the biggest impediments to practical consumption of XAI is the inherent difficulty to evaluate if a given explanation is correct or not. For post hoc explanation methods, the usual metric is fidelity, namely how accurately the surrogate model built by the explanation method approximates the true model (Plumb et al., 2018). In fact, LIME is explicitly constructed around a trade-off between fidelity and the complexity of the resulting explantion (Ribeiro et al., 2016); other recent work generalizes the purpose of explanations as a multiobjective game with trade-offs between explanatory accuracy, simplicity and relevance (Watson and Floridi, 2020). However, fidelity is measured over the entire possible domain of inputs to a model and therefore places undue emphasis on unobserved and infeasible parts of the input space (Frye et al., 2019). Apart from such quantitative assessments of correctness, which are not free of problems, the best we can do is appeal to formal philosophical notions of epistemology (Mittelstadt et al., 2019), but at the cost of any quantification of explanatory accuracy.

Challenge 5.

XAI techniques in general lack robustness and have strong basis dependence.

Local feature importance methods SHAP and LIME are known to not be robust (Alvarez-Melis and Jaakkola, 2018; Slack et al., 2020). Several studies have demonstrated that Shapley-value based methods like SHAP suffer from an effect similar in spirit to multicollinearity, assigning spuriously low importances to highly dependent features (Hooker and Mentch, 2019). In the extreme case of identical features, the result Shapley values is effectively averaged over each feature, thus resulting in artificially lowered feature importances (Kumar et al., 2020). In addition, there are subtle dependencies of Shapley value-based methods on the data distribution (Frye et al., 2019; Chen et al., 2020; Sundararajan and Najmi, 2020). Similar results exists for other techniques such as influence functions, which arise from nonconvex effects (Basu et al., 2021). Such results caution that the basis of features has to be carefully considered in crafting an explanation.

Challenge 6.

Feature importance explanation methods can be manipulated.

Recent work on “fairwashing” has demonstrated that feature importance methods can be easily manipulated to create arbitrary feature importance rankings, by artificially varying the behavior of the model on unobserved regions of the input space (Dombrowski et al., 2019; Aïvodji et al., 2019; Anders et al., 2020; Slack et al., 2020; Dimanov et al., 2020). Such results argue in favor of developing new explanation methods that are robust against such manipulation (Alvarez-Melis and Jaakkola, 2018; Hancox-Li, 2020). Initial research in this direction around robust optimization are promising (Kulynych et al., 2020), but their usefulness in practice remains to be seen.

Challenge 7.

Too detailed an explanation can compromise a proprietary model that was intended to be kept confidential.

Research has shown that the output of feature importance methods like SHAP, through repeated queries, are prone to membership attacks that can reveal intimate details about the classification boundary (Shokri et al., 2019). Similar research has shown that counterfactual explanations are vulnerable to similar attacks (Aïvodji et al., 2020), as are image-based explanations like saliency maps (Zhao et al., 2021). Such results reveal the risk that when providing explanations to external stakeholders, the recipients of such explanations can collude to reconstruct the inner workings of a model. Such information leakage is not just theoretical, but has been realized such as the public discovery of the Chase 5/24 rule, simply by comparing decision outcomes across multiple applicants (Kerr, 2021).

3. Conclusions and outlook

We have organized the XAI literature into seven distinct challenges for providing a unified thesis for specifying the appropriate use of XAI techniques to handle the myriad use cases in industry. Numerous recent papers conclude that details such as the specific intent and needs of multiple stakeholders, global or local scope of desired explanation, and type of data and model to be explained, be they image classifiers, speech recognition engines, adaptive chatbots, or recommender systems matter greatly and argue in favor of customized techniques for each application. Furthermore, the lack of uniform evaluation criteria for verifying the correctness of explanations necessitate a nuanced consideration of how humans consumer and react to explanations provided to them, and results showing tendencies to overtrust AI-generated explanations ought to be factored in when crafting explainability guidelines for AI systems. The innate brittleness of existing XAI techniques means that they are vulnerable to malicious manipulation to produce misleading evidence for reassuring overtrusting humans. Finally, the desire for explanability needs to be balanced against other competing needs such as privacy and security, or else risk compromising the original AI system’s intended purpose or revealing details of a proprietary model that was intended to be kept confidential. Our summary of the literature reveals these common themes of real world challenges, thus meriting a cautious and principled approach for using XAI in the full appreciation of the specific context of stakeholder needs, business use case, and details of the AI system’s construction.


This paper was prepared for informational purposes by the Artificial Intelligence Research group of JPMorgan Chase & Co and its affiliates (“JP Morgan”), and is not a product of the Research Department of JP Morgan. JP Morgan makes no representation and warranty whatsoever and disclaims all liability, for the completeness, accuracy or reliability of the information contained herein. This document is not intended as investment research or investment advice, or a recommendation, offer or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute a solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction or to such person would be unlawful. © 2021 JPMorgan Chase & Co. All rights reserved.


  • [1] Cited by: §2.
  • M. Abdar, F. Pourpanah, S. Hussain, D. Rezazadegan, L. Liu, M. Ghavamzadeh, P. Fieguth, X. Cao, A. Khosravi, U. R. Acharya, V. Makarenkov, and S. Nahavandi (2020) A review of uncertainty quantification in deep learning: Techniques, applications and challenges. arXiv. External Links: 2011.06225, ISSN 23318422 Cited by: §2.
  • U. Aïvodji, H. Arai, O. Fortineau, S. Gambs, S. Hara, and A. Tapp (2019) Fairwashing: the risk of rationalization. In Proceedings of the 36th International Conference on Machine Learning, pp. 161–170. External Links: 1901.09749, Link Cited by: §2.
  • U. Aïvodji, A. Bolot, and S. Gambs (2020) Model extraction from counterfactual explanations. arXiv. External Links: 2009.01884, ISSN 23318422 Cited by: §2.
  • E. Albini, P. Lertvittayakumjorn, A. Rago, and F. Toni (2020)

    DAX: Deep Argumentative eXplanation for Neural Networks

    External Links: 2012.05766, Link Cited by: §2.
  • D. Alvarez-Melis and T. S. Jaakkola (2018) On the Robustness of Interpretability Methods. In Proceedings of the 2018 ICML Workshop on Human Interpretability in Machine Learning, External Links: 1806.08049, Link Cited by: §2, §2.
  • D. Amir and O. Amir (2018) HIGHLIGHTS: Summarizing agent behavior to people. Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS 2, pp. 1168–1176. External Links: ISBN 9781510868083, ISSN 15582914 Cited by: §2.
  • C. J. Anders, P. Pasliev, A. Dombrowski, K. Müller, and P. Kessel (2020) Fairwashing Explanations with Off-Manifold Detergent. In Proceedings of the 37th International Conference on Machine Learning, pp. 314–323. External Links: 2007.09969, Link Cited by: §2.
  • A. Artelt and B. Hammer (2019) On the computation of counterfactual explanations – A survey. External Links: 1911.07749, Link Cited by: §2.
  • S. Basu, P. Pope, and S. Feizi (2021) Influence functions in deep learning are fragile. pp. 1–22. External Links: 2006.14651v2 Cited by: §2.
  • J. Bennett (2003) A philosophical guide to conditionals. Clarendon Press, Oxford, UK. Cited by: §2.
  • L. Bertossi, J. Li, M. Schleich, D. Suciu, and Z. Vagena (2020) Causality-based Explanation of Classification Outcomes. Proceedings of the 4th Workshop on Data Management for End-To-End Machine Learning, DEEM 2020 - In conjunction with the 2020 ACM SIGMOD/PODS Conference. External Links: Document, 2003.06868, ISBN 9781450380232 Cited by: §2.
  • U. Bhatt, M. Andrus, A. Weller, and A. Xiang (2020) Machine Learning Explainability for External Stakeholders. External Links: 2007.05408, Link Cited by: §2.
  • Board of Governers of the Federal Reserve System and Office of the Comptroller of the Currency (2011) Supervisory guidance on model risk management. Technical report Technical Report OCC 2011-12 / SR 11-7. Cited by: §1.
  • R. M.J. Byrne (2016) Counterfactual thought. Annual Review of Psychology 67, pp. 135–157. External Links: Document, ISSN 15452085 Cited by: §2.
  • M. Cashmore, A. Collins, B. Krarup, S. Krivic, D. Magazzeni, and D. Smith (2019) Towards Explainable AI Planning as a Service. External Links: 1908.05059, Link Cited by: §2.
  • H. Chen, J. D. Janizek, S. Lundberg, and S. Lee (2020) True to the Model or True to the Data?. (3). External Links: 2006.16234, Link Cited by: §2.
  • J. Chen (2018) Fair lending needs explainable models for responsible recommendation. pp. 4. External Links: Document, 1809.04684, ISBN 9781450361729, Link Cited by: §1, §1.
  • P. Cousot and R. Cousot (1992)

    Abstract interpretation and application to logic programs

    The Journal of Logic Programming 13 (2-3), pp. 103–179. External Links: Document, ISSN 07431066, Link Cited by: §2.
  • S. Dasgupta, N. Frost, M. Moshkovitz, and C. Rashtchian (2020) Explainable -Means and -Medians Clustering. In Proceedings of the 37th International Conference on Machine Learning, pp. 7055–7065. External Links: 2002.12538, Link Cited by: §2.
  • A. Deva, S. Shingi, A. Tiwari, N. Bannur, S. Jain, J. White, A. Raval, and S. Merugu (2021) Interpretability of epidemiological models: the curse of non-identifiability. pp. 1–8. External Links: 2104.14821 Cited by: §2.
  • B. Dimanov, U. Bhatt, M. Jamnik, and A. Weller (2020) You shouldn’t trust me: Learning models which conceal unfairness from multiple explanation methods. Frontiers in Artificial Intelligence and Applications 325, pp. 2473–2480. External Links: Document, ISBN 9781643681009, ISSN 09226389 Cited by: §2.
  • A. K. Dombrowski, M. Alber, C. J. Anders, M. Ackermann, K. R. Müller, and P. Kessel (2019) Explanations can be manipulated and geometry is to blame. arXiv (NeurIPS 2019), pp. 1–12. External Links: 1906.07983, ISSN 23318422 Cited by: §2.
  • U. Ehsan, Q. V. Liao, M. Muller, M. O. Riedl, and J. D. Weisz (2021a) Expanding explainability: towards social transparency in ai systems. New York, NY, USA. External Links: ISBN 9781450380966, Link, Document Cited by: §2.
  • U. Ehsan and M. O. Riedl (2020) Human-Centered Explainable AI: Towards a Reflective Sociotechnical Approach. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 12424 LNCS, pp. 449–466. External Links: Document, 2002.01092, ISBN 9783030601164, ISSN 16113349 Cited by: §2.
  • U. Ehsan, P. Wintersberger, Q. V. Liao, M. Mara, M. Streit, S. Wachter, A. Riener, and M. O. Riedl (2021b) Operationalizing Human-Centered Perspectives in Explainable AI. In ACM CHI Workshop on Operationalizing Human-Centered Perspectives in Explainable AI, pp. 9. External Links: ISBN 9781450380959 Cited by: §2.
  • C. Frye, I. Feige, and C. Rowat (2019) Asymmetric Shapley values: incorporating causal knowledge into model-agnostic explainability. (NeurIPS). External Links: 1910.06358, Link Cited by: §2, §2, §2.
  • B. Ghai, Q. V. Liao, Y. Zhang, R. Bellamy, and K. Mueller (2021) Explainable active learning (xal): toward ai explanations as interfaces for machine teachers. Proc. ACM Hum.-Comput. Interact. 4 (CSCW3). External Links: Link, Document Cited by: §2.
  • E. Guerra, J. de Lara, A. Malizia, and P. Díaz (2009) Supporting user-oriented analysis for multi-view domain-specific visual languages. Information and Software Technology 51 (4), pp. 769–784. External Links: Document, ISBN 0950-5849, ISSN 09505849, Link Cited by: §2.
  • L. Hancox-Li (2020) Robustness in machine learning explanations: Does it matter?. In FAT* 2020 - Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 640–647. External Links: Document, ISBN 9781450369367 Cited by: §2.
  • W. L. Harper, R. Stalnaker, and G. Pearce (Eds.) (1981) Ifs: Conditionals, Belief, Decision, Chance, and Time. The University of Western Ontario Series in Philosophy of Science 15, D. Reidel, Dordrecht, Holland. External Links: Document Cited by: §2.
  • G. Hooker and L. Mentch (2019) Please stop permuting features an explanation and alternatives. arXiv, pp. 1–15. External Links: 1905.03151, ISSN 23318422 Cited by: §2.
  • E. Horel, K. Giesecke, V. Storchan, and N. Chittar (2019) Explainable Clustering and Application to Wealth Management Compliance. pp. 2–7. External Links: 1909.13381, Link Cited by: §2.
  • M. Hurley and J. Adebayo (2016) Credit scoring in the era of big data. Yale Journal of Law & Technology 148, pp. 148–216. Cited by: §1.
  • D. Jannach and M. Jugovac (2019) Measuring the business value of recommender systems. ACM Trans. Manage. Inf. Syst. 10 (4). External Links: ISSN 2158-656X, Link, Document Cited by: §1.
  • S. Jesus, C. Belém, V. Balayan, J. Bento, P. Saleiro, P. Bizarro, and J. Gama (2021) How can I choose an explainer? An Application-grounded Evaluation of Post-hoc Explanations Sérgio. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, New York, NY, USA, pp. 805–815. External Links: Document, 2101.08758, ISBN 9781450383097, Link Cited by: §2.
  • S. Kambhampati (2019) Synthesizing explainable behavior for human-ai collaboration. Richland, SC, pp. 1–2. External Links: ISBN 9781450363099 Cited by: §2.
  • M. E. Kaminski and G. Malgieri (2020) Multi-layered explanations from algorithmic impact assessments in the GDPR. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, New York, NY, USA, pp. 68–79. External Links: Document, ISBN 9781450369367, Link Cited by: §2.
  • D. Kang, D. Raghavan, P. Bailis, M. Zaharia, and S. Dawn (2018) Model Assertions for Debugging Machine Learning. In NIPS2018 workshop, pp. 9. External Links: Link Cited by: §2.
  • A. H. Karimi, B. Schölkopf, and I. Valera (2020) Algorithmic recourse: From counterfactual explanations to interventions. arXiv, pp. 353–362. External Links: 2002.06278, ISBN 9781450383097, ISSN 23318422 Cited by: §2.
  • A. Kasirzadeh (2019) Mathematical decisions and non-causal elements of explainable AI. (NeurIPS), pp. 1–100. External Links: 1910.13607, Link Cited by: §2.
  • H. Kaur, H. Nori, S. Jenkins, R. Caruana, H. Wallach, and J. Wortman Vaughan (2020) Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning. CHI Conference on Human Factors in Computing Systems Proceedings, pp. 1–14. External Links: Document, ISBN 9781450367080, Link Cited by: §2.
  • R. Kerr (2021) Chase’s 5/24 rule: everything you need to know. The Points Guy. External Links: Link Cited by: §2.
  • P. J. Kindermans, S. Hooker, J. Adebayo, M. Alber, K. T. Schütt, S. Dähne, D. Erhan, and B. Kim (2019) The (Un)reliability of Saliency Methods. In Lecture Notes in Computer Science, Vol. 11700 LNCS, pp. 267–280. External Links: Document, 1711.00867, ISSN 16113349 Cited by: §2.
  • B. Kulynych, R. Overdorf, C. Troncoso, and S. Gürses (2020) POTS: protective optimization technologies. FAT* 2020 - Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 177–188. External Links: Document, 1806.02711, ISBN 9781450369367 Cited by: §2.
  • I. E. Kumar, S. Venkatasubramanian, C. Scheidegger, and S. Friedler (2020) Problems with Shapley-value-based explanations as feature importance measures. In Proceedings of the 37th International Conference on Machine Learning, Vol. 119, pp. 5491–5500. External Links: 2002.11097, Link Cited by: §2, §2.
  • E. Kurshan, H. Shen, and J. Chen (2020) Towards self-regulating ai: challenges and opportunities of ai model governance in financial services. New York, NY, USA. External Links: 2010.04827 Cited by: §1.
  • M. Lecuyer, V. Atlidakis, R. Geambasu, D. Hsu, and S. Jana (2019) Certified robustness to adversarial examples with differential privacy. In 2019 IEEE Symposium on Security and Privacy (SP), pp. 656–673. External Links: Document Cited by: §2.
  • S. M. Lundberg, G. G. Erion, and S. Lee (2018) Consistent Individualized Feature Attribution for Tree Ensembles. (2). External Links: 1802.03888, Link Cited by: §2.
  • S. M. Lundberg and S. I. Lee (2017) A unified approach to interpreting model predictions. Advances in Neural Information Processing Systems 2017-Decem (Section 2), pp. 4766–4775. External Links: 1705.07874, ISSN 10495258, Link Cited by: §2.
  • R. Luss, P. Chen, A. Dhurandhar, P. Sattigeri, Y. Zhang, K. Shanmugam, and C. Tu (2019) Generating Contrastive Explanations with Monotonic Attribute Functions. pp. 1–26. External Links: 1905.12698, Link Cited by: §2.
  • A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu (2018) Towards deep learning models resistant to adversarial attacks. 6th International Conference on Learning Representations, ICLR 2018 - Conference Track Proceedings, pp. 1–28. External Links: 1706.06083 Cited by: §2.
  • P. Madumal, T. Miller, L. Sonenberg, and F. Vetere (2020) Distal Explanations for Explainable Reinforcement Learning Agents. External Links: 2001.10284, Link Cited by: §2.
  • T. Miller (2019) Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence 267, pp. 1–38. External Links: Document, 1706.07269, ISSN 00043702, Link Cited by: §2, §2.
  • B. Mittelstadt, C. Russell, and S. Wachter (2019) Explaining explanations in AI. In FAT* 2019 - Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency, pp. 279–288. External Links: Document, 1811.01439, ISBN 9781450361255 Cited by: §2.
  • R. K. Mothilal, A. Sharma, and C. Tan (2020) Explaining machine learning classifiers through diverse counterfactual explanations. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, New York, NY, USA, pp. 607–617. External Links: Document, 1905.07697, ISBN 9781450369367, Link Cited by: §2.
  • J. Mullenbach, S. Wiegreffe, J. Duke, J. Sun, and J. Eisenstein (2018) Explainable prediction of medical codes from clinical text. New Orleans, Louisiana, pp. 1101–1111. External Links: Link, Document Cited by: §2.
  • M. Neely, S. F. Schouten, M. J. R. Bleeker, and A. Lucic (2021) Order in the Court: Explainable AI Methods Prone to Disagreement. External Links: 2105.03287v1 Cited by: §2.
  • New York Department of Financial Services (2021) DFS issues findings on the apple card and its underwriter goldman sachs bank. External Links: Link Cited by: §1.
  • E. Ntoutsi, P. Fafalios, U. Gadiraju, V. Iosifidis, W. Nejdl, M. Vidal, S. Ruggieri, F. Turini, S. Papadopoulos, E. Krasanakis, I. Kompatsiaris, K. Kinder-Kurlanda, C. Wagner, F. Karimi, M. Fernandez, H. Alani, B. Berendt, T. Kruegel, C. Heinze, K. Broelemann, G. Kasneci, T. Tiropanis, and S. Staab (2020) Bias in Data-driven AI Systems – An Introductory Survey. pp. 1–19. External Links: 2001.09762, Link Cited by: §2.
  • N. Y. S. D. of Financial Services (2021) Report on apple card investigation. Technical report New York State Department of Financial Services, New York, NY. External Links: Link Cited by: §1.
  • J. Pearl (2009) Causality. 2 edition. External Links: ISBN 9780521895606 Cited by: §2.
  • G. Plumb, D. Molitor, and A. Talwalkar (2018) Model agnostic supervised local explanations. In Proceedings of the 32nd Conference on Neural Information Processing Systems, pp. 2520–2529. External Links: Document, ISSN 23318422 Cited by: §2.
  • J. L. Pollock (1976) Subjunctive conditionals. 1 edition, Philosophical Studies Series 08, D. Reidel Publishing Company, Dordrecht, Holland; Boston, USA. External Links: Document Cited by: §2.
  • G. Ras, M. van Gerven, and P. Haselager (2018) Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges. In Explainable and Interpretable Models in Computer Vision and Machine Learning, The Springer Series on Challenges in Machine Learning, pp. 19–36. External Links: Document, 1803.07517, ISSN 23318422, Link Cited by: §2.
  • M. T. Ribeiro, S. Singh, and C. Guestrin (2016) ”Why Should I Trust You?”: Explaining the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD ’16, New York, New York, USA, pp. 1135–1144. External Links: Document, 1602.04938, ISBN 9781450342322, Link Cited by: §2, §2.
  • T. Samp, S. R. Phillips, and I. E. Sanchez (2021) Technology, artificial intelligence in focus for the biden administration and the 117th congress: seen through the lens of competition with china. DLA Piper AI Outlook. External Links: Link Cited by: §1.
  • M. H. Sarhan, A. Eslami, N. Navab, and S. Albarqouni (2019) Learning interpretable disentangled representations using adversarial vaes. Cham, pp. 37–44. External Links: ISBN 978-3-030-33391-1 Cited by: §2.
  • R. Shokri, M. Strobel, and Y. Zick (2019) On the Privacy Risks of Model Explanations. External Links: 1907.00164, Link Cited by: §2.
  • M. A. Siddiqui, J. W. Stokes, C. Seifert, E. Argyle, R. McCann, J. Neil, and J. Carroll (2019)

    Detecting cyber attacks using anomaly detection with explanations and expert feedback

    pp. 2872–2876. External Links: Document Cited by: §1.
  • D. Slack, S. Hilgard, E. Jia, S. Singh, and H. Lakkaraju (2020) Fooling LIME and SHAP: Adversarial attacks on post hoc explanation methods. AIES 2020 - Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pp. 180–186. External Links: Document, 1911.02508, ISBN 9781450371100 Cited by: §2, §2.
  • I. Stepin, J. M. Alonso, A. Catala, and M. Pereira-Farina (2021) A Survey of Contrastive and Counterfactual Explanation Generation Methods for Explainable Artificial Intelligence. IEEE Access 9, pp. 11974–12001. External Links: Document, ISSN 21693536 Cited by: §2.
  • M. Sundararajan and A. Najmi (2020) The many Shapley values for model explanation. In Proceedings of the 37th International Conference on Machine Learning, pp. 9269–9278. External Links: 1908.08474, Link Cited by: §2.
  • The Comptroller of the Currency, The Federal Reserve System, The Federal Deposit Insurance Corporation, The Consumer Financial Protection Bureau, and The National Credit Union Administration (2021) Request for information and comment on financial institutions’ use of artificial intelligence, including machine learning. Federal Register, pp. 16837–16842. Cited by: §1.
  • R. Tomsett, D. Braines, D. Harborne, A. Preece, and S. Chakraborty (2018) Interpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systems. External Links: 1806.07552, Link Cited by: §2.
  • B. Ustun, A. Spangher, and Y. Liu (2019) Actionable Recourse in Linear Classification. In Proceedings of the Conference on Fairness, Accountability, and Transparency, New York, NY, USA, pp. 10–19. External Links: Document, 1809.06514, ISBN 9781450361255, Link Cited by: §2.
  • J. Vig, W. Kryscinski, K. Goel, and N. F. Rajani (2021)

    SummVis: Interactive Visual Analysis of Models, Data, and Evaluation for Text Summarization

    External Links: 2104.07605, Link Cited by: §2.
  • M. Virgolin, A. De Lorenzo, F. Randone, E. Medvet, and M. Wahde (2021)

    Model Learning with Personalized Interpretability Estimation (ML-PIE)

    External Links: 2104.06060, Link Cited by: §2.
  • Y. Wang and V. Rockova (2020) Uncertainty quantification for sparse deep learning. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, S. Chiappa and R. Calandra (Eds.), Proceedings of Machine Learning Research, Vol. 108, pp. 298–308. External Links: Link Cited by: §2.
  • D. S. Watson and L. Floridi (2020) The explanation game: a formal framework for interpretable machine learning. Synthese. External Links: Document, ISBN 1122902002629, ISSN 0039-7857, Link Cited by: §2.
  • J. Wawrzinek, S. A. R. Hussaini, O. Wiehr, J. M. G. Pinto, and W. Balke (2020) Explainable word-embeddings for medical digital libraries - a context-aware approach. In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)InterspeechDomain Adaptation and Representation Transfer and Medical Image Learning with Less Labels and Imperfect DataProceedings of the 19th ACM International Conference on Intelligent Virtual AgentsProceedings of the Conference on Computer Vision and Pattern RecognitionProceedings of the AAAI Conference on Artificial Intelligence, Ethics and SocietyProceedings of the 2021 CHI Conference on Human Factors in Computing SystemsProceedings of the International Conference on Learning RepresentationsProceedings of the International Conference on Learning RepresentationsProceedings of the International Conference on Learning RepresentationsProceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System DemonstrationsProceedings of the 18th International Conference on Autonomous Agents and MultiAgent SystemsICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)Proceedings of the 1st ACM International Conference on AI in Finance2nd FATREC Workshop on Responsible Recommendation, Q. Wang, F. Milletari, H. V. Nguyen, S. Albarqouni, M. J. Cardoso, N. Rieke, Z. Xu, K. Kamnitsas, V. Patel, B. Roysam, S. Jiang, K. Zhou, K. Luu, and N. Le (Eds.), JCDL ’20IVA ’19CHI ’21AAMAS ’19, Vol. , New York, NY, USA. External Links: ISBN 9781450375856, Link, Document Cited by: §2.
  • M. Weber, D. K. I. Weidele, G. Domeniconi, C. Bellei, C. E. Leiserson, J. Chen, and T. Robinson (2019) Anti-money laundering in bitcoin: experimenting with graph convolutional networks for financial forensics. arXiv (10). External Links: 1908.02591, ISSN 23318422 Cited by: §1.
  • K. Weitz, D. Schiller, R. Schlagowski, T. Huber, and E. André (2019) ”Do you trust me?”: increasing user-trust by integrating virtual agents in explainable ai interaction design. New York, NY, USA, pp. 7–9. External Links: ISBN 9781450366724, Link, Document Cited by: §2.
  • N. Xie, G. Ras, M. van Gerven, and D. Doran (2020) Explainable Deep Learning: A Field Guide for the Uninitiated. External Links: 2004.14545, Link Cited by: §2.
  • K. Xu, H. Zhang, S. Wang, Y. Wang, S. Jana, X. Lin, and C. J. Hsieh (2020) Fast and complete: Enabling complete neural network verification with rapid and massively parallel incomplete verifiers. arXiv, pp. 1–15. External Links: 2011.13824, ISSN 23318422 Cited by: §2.
  • Y. Yang, E. Kandogan, Y. Li, W. S. Lasecki, and P. Sen (2019) HEIDL: Learning linguistic expressions with deep learning and human-in-the-loop. pp. 135–140. External Links: ISSN 23318422 Cited by: §2.
  • G. Yona, A. Ghorbani, and J. Zou (2021) Who’s responsible? Jointly quantifying the contribution of the learning algorithm and training data. pp. 1–23. External Links: 1910.04214, ISSN 23318422 Cited by: §2.
  • Y. Zhang and X. Chen (2020) Explainable recommendation: a survey and new perspectives. Foundations and Trends® in Information Retrieval 14 (1), pp. 1–101. External Links: Document, ISSN 1554-0669, Link Cited by: §2.
  • X. Zhao, W. Zhang, X. Xiao, and B. Y. Lim (2021) Exploiting Explanations for Model Inversion Attacks. External Links: 2104.12669, Link Cited by: §2.
  • Y. Zhao, X. Xu, and M. Wang (2019) Predicting overall customer satisfaction: big data evidence from hotel online textual reviews. International Journal of Hospitality Management 76, pp. 111–121. External Links: Document, ISSN 0278-4319, Link Cited by: §1.
  • X. Zhu, C. Xu, and D. Tao (2021) Where and What? Examining Interpretable Disentangled Representations. External Links: 2104.05622, Link Cited by: §2.