The interpretation of data-driven models explains their input-output relationship, which provides information about whether the models admit some undesired characteristics, and thus can guide people to use, debug, and improve machine learning models. The model interpretation has an increasing demand in many real-applications, including medicine(Wang et al., 2019), security (Chakraborti et al., 2019), and criminal justice (Lipton, 2018).
Existing research on model interpretation can be categorized into model-specific methods and model-agnostic methods. Model-specific methods take advantage of the knowledge of the model itself to assist explanations, such as gradient-based methods for neural networks, whereas model-agnostic methods can explain any black-box system. Instance-wise Feature Selection (IFS) is a well known model-agnostic interpretation method. It produces an importance score of each feature for representing a data sample (Du et al., 2019), which indicates how much each feature dominates the model’s output. For this kind of approach, desired properties for ideal explanations (feature importance scores) are as follows.
Recent research for IFS-based model explanation can be divided into (local/global) feature attribution methods (Ancona et al., 2017; Yeh et al., 2019)111In this paper, the definitions of global and local explanations follow the description of Ancona et al. (Ancona et al., 2017) and Yeh et al. (Yeh et al., 2019), and distinct from that of Plumb et al. (Plumb et al., 2018). and direct model-interpretation (DMI) methods. Local feature attribution methods provide some sensitivity scores of the model output concerning the changes of the features in the neighborhood. In contrast, global feature attribution methods directly produce the amount of change of the model output given changes of the features. Other than providing the change of the model output, DMI is a more straightforward approach to select features and use a model to approximate the output of the original black-box model (Chen et al., 2018; Sundararajan et al., 2017).
In this paper, we attempt to tackle the DMI problem. When given a data sample and the model to be explained, what features does the model use primarily to generate the output? A straightforward approach is to develop a feature attribution network (which we refer to as the explainer) to produce a soft/hard mask to highlight essential features, and a prediction network (which we refer to as the approximator) to approximate the output of the original model (Chen et al., 2018). However, this straightforward approach may be faced with four following problems.
Sanity problem (Adebayo et al., 2018b): a mask may be irrelevant to the original model, but only relate to the features of a specific sample. As a consequence, the selected features of the trained explainer may be different with those truly used by the original model, which is not expected in interpreting the model.
combinatorial shortcuts problem: the entries of the mask may not select good features, but rather act as additional features themselves for better approximation performances (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019), because it is a function of all the input features. For example, the explainer could choose to mask out the first half of the features for positive samples, and the second half of the features for negative samples. The approximator can utilize this pattern to predict the target while completely ignore whether good features are selected.
Model identifiability problem: similar approximation performances can be achieved by different groups of feature. It is difficult to decide which group is the best.
Information transmission problem (Park et al., 2019): it is difficult to transmit effective supervised information to the explainer, because the mask is unsupervised.
To address these issues, we propose a Model-agnostic Effective Efficient Direct (MEED) model-interpretation method for addressing the DMI problem. The overall architecture of our proposed framework is presented in Figure 1. The major components include model output feedback, adversarial infidelity learning, and prior knowledge-based warm start, which we describe as follows.
Firstly, we propose to enrich the input information of the explainer to boost the effectiveness and the efficiency of the feature selection process. Existing research treats raw features only as the input to the neural network-based explainer (Schwab and Karlen, 2019; Chen et al., 2018; Bang et al., 2019). The absence of the original model to include for the explainer’s input may render the mask out of the explainer uncorrelated with the original model and then cause the sanity problem. Nonetheless, it is not trivial to input a whole model into the neural network-based explainer. Therefore, we propose to incorporate the model output as another input signal. Apart from the sanity problem, the model output can provide rich information for the explainers to select essential features, and make the learning process more precise, especially in applications like regression or representation learning. In other words, the information transmission problem can also be mitigated.
Secondly, we propose to exploit the unselected features for mitigating the combinatorial shortcuts and model identifiability problems. Inspired by Hooker et al. (Hooker et al., 2019), we attempt to achieve an auxiliary goal that the unselected features should contain the least useful information. To achieve this, we propose an Adversarial Infidelity Learning (AIL) mechanism. Specifically, we develop another approximator that learns to approximate the original model output using the unselected features. Then our explainer learns to select features to minimize such approximation accuracy. The learning processes run alternately. Intuitively, the convergence of such an adversarial learning process will render the masks uncorrelated with the model output, and then can mitigate the combinatorial shortcuts problems. On the other hand, this learning process exploits the unselected features, which are often (at least relatively) ignored, to introduce additional supervised information for a certain group of selected features, and then can improve model identifiability. These properties are demonstrated by our theoretical analysis and experimental results.
Finally, we extend our framework to further mitigate the information transmission problem by integrating prior knowledge. Specifically, we integrate explanations provided by efficient interpretation methods as priors to provide a warm start. The constraints of the priors fade out when the number of training epochs grows to learn a more powerful explainer by the end-to-end framework.
We follow Chen (Chen et al., 2018) to perform a predictive evaluation to see whether the selected features contribute to sufficient approximate accuracy. Comprehensive empirical evaluation results on four real-world benchmark datasets are provided with quantitative evaluation metrics and human-evaluations to demonstrate the effectiveness and superiority of our proposed method. Moreover, we validate our method on a real-world application: teenager/adult classification based on mobile sensor data from million of Tencent users who play the popular Honor of Kings, a.k.a. Arena of Valor game.
2. Related Works
Model interpretation methods based on IFS can be categorized into local/global methods as introduced in the introduction. Local methods includes 1) gradient-based methods, such as Gradient (Grad) (Simonyan et al., 2013) and Guided Back-Propagation (Springenberg et al., 2015), 2) sampling-based methods, i.e., perform sensitivity analysis by sampling points around the given data sample, such as LIME (Ribeiro et al., 2016), kernel SHAP (Lundberg and Lee, 2017) and CXPlain (Schwab and Karlen, 2019), and 3) hybrid methods, such as SmoothGrad (Smilkov et al., 2017), Squared SmoothGrad (Smilkov et al., 2017), VarGrad (Adebayo et al., 2018a), and INFD (Yeh et al., 2019). On the other hand, global methods include Gradient Input (Shrikumar et al., 2017), Integrated Gradients (Sundararajan et al., 2017), DeepLIFT (Shrikumar et al., 2017) and LRP (Bach et al., 2015), among others. These methods do not directly tackle the DMI problem.
For the DMI problem, being inherently interpretable, tree- (Schwab and Hlavacs, 2015) and rule-based (Andrews et al., 1995) models have been proposed to approximate the output of a complex black-box model with all features. The models themselves provide explanations, including feature importance. However, they may lack the ability for accurate approximations when the original given model is complex. Recently, L2X (Chen et al., 2018) and VIBI (Bang et al., 2019)
have been proposed as variational methods to learn a neural network-based approximator based on the selected features. The unselected features are masked out by imputing zeros. VIBI improves L2X to encourage the briefness of the learned explanation by adding a constraint for the feature scores to a global prior (in contrast, our priors are conditioned on each sample). Since L2X and VIBI only input features to their explainers, and they directly select features to approximate the model out, therefore, they both may suffer from thesanity, combinatorial shortcuts, model identifiability, model identifiability, and information transmission problems.
In contrast, our method tackle these problems for DMI by leveraging more comprehensive information from the model output, the proposed adversarial infidelity learning mechanism, and the proposed prior-knowledge integration. We note that Zhu et al. (Zhu et al., 2019) proposed an adversarial attention network. However, their objective is to eliminate the difference in extracted features for different learning tasks, which is different from ours.
In this section, we present the detailed methodology of our method. First, we define the notations and problem settings of our study.
Consider a dataset consisting of independent samples. For the th sample,
is a feature vector withdimensions, is the output vector of a given data-driven model (note that may be different from the true label of the sample). The conditional output distribution is determined by the given model. For classification tasks, is the number of classes.
We do not assume the true label of each feature vector is available for training or inference. We develop a neural network-based IFS explainer , which outputs a feature-importance-score vector for a data sample and the model . As discussed by Yeh et al. (Yeh et al., 2019), the explainer should be a mapping that . Since it is not trivial to treat an arbitrary model in as an input to a neural network, we compromise by involving the model output as an alternative such that . We select top- features according to , where is a user-defined parameter. The indices of selected features are denoted by . For a feature vector , the selected features are denoted by , whereas the unselected features are denoted by . Throughout the paper, we denote as the index set for some integer .
The goal of our learning approach is to train a neural network-based explainer over the dataset and then generalize it to a testing set to see whether the selected features contribute to sufficient approximate accuracy. The quantitative evaluations of the explainer are described in Section 4.0.2.
3.1. Our Framework
The architecture of our framework is illustrated in Fig. 1. We explain a given model by providing IFS for each specific data sample. The IFS is embodied as a feature attribution mask provided by a learned explainer with the features and the model output of the data sample as inputs. We train an approximator to use selected/masked features to approximate the model output. We also train an adversarial approximator to use unselected features to approximate the model output, and then train the explainer to select features to undermine the approximation, which is referred to as the AIL mechanism. As an extention, integrating efficient model-interpretation methods is also introduced to provide a warm start.
3.2. Adversarial Infidelity Learning
As discussed in the introduction, a straightforward approach to optimize the selection indices is directly maximizing the mutual information between selected features , and the model output (Chen et al., 2018; Bang et al., 2019). To tackle the combinatorial shortcuts and model identifiability problems, we propose an auxiliary objective: minimizing the mutual information between unselected features and the model output . Because compared with the selected features, the unselected features should contain less useful information. Therefore, the basic optimization problem for is:
We can be guided by the Theorem 1 to optimize the explainer.
Theorem 1 ().
According to Theorem 1, we deveop an approximator to learn the conditional distribution . We achieve this by optimizing a variational mapping: to let approximate . We define , where
denotes the loss function corresponding to the conditional distribution(e.g.,
mean square error for Gaussian distribution, and categorical cross entropy for categorical distribution), andwhich is defined as: if and otherwise. We let denote the output distribution of . We approximate by instead of , because as discussed by Hooker et al. (Hooker et al., 2019), , then may approximate more accurate than does.
Similarly, we develop another approximator to learn to approximate . Then we can show that Problem (1) can be relaxed by maximizing variational lower bounds and alternately optimizing:
First, Problem (3) is optimized to learn and to approximate and , respectively. Then Problem (4) is optimized to learn the explainer to find good explanations according to Theorem 1. Since 1) is maximized by optimizing and then minimized by optimizing , which is an adversarial learning process, and 2) minimizing represents infidelity, i.e., undermining performance to approximate (by excluding selected features), the alternate optimization process can be regarded as an adversarial infidelity learning mechanism.
Since optimizing Problems (3) and (4) for all possible requires times of computation for the objectives, we follow L2X (Chen et al., 2018) to apply the Gumbel-softmax trick to approximately sample a -hot vector. Specifically, let for a pair of inputs , where for , and . Then we define the sampled vector , where for a predefined ,
Denote the above random mapping by , approximate and , where with all elements being , and denotes element-wise product. Define the losses for selected and unselected features, respectively,
For inference, one can select the top- features of for a testing sample , where .
3.3. Theoretical Analysis
In this section, we first derive variational lower bounds to show the connection between the goal in Eq. (1) and the realization in Eq. (3) and (4). The derivation also shows that the approximator is possible to be superior to the given model to predict the model output using masked features. We also show that our AIL mechanism can mitigate the combinatorial shortcuts problem.
The variational lower bounds are as follows and derived in Appendix A.2. First, for selected features, we have for any :
where the equality holds if and only if . Therefore, if ’s output distribution , it is possible that , i.e., can be more accurate than
Similarly, for unselected features, we have for any :
On the other hand, since actually receives the selected features and the feature-attribution mask as inputs, where , what actually learns is the conditional distribution . Through the straightforward learning mentioned in the introduction, i.e., removing in Eq. (7) and (8), it could cause the combinatorial shortcuts problem for to learn only, resulting in the feature selection process meaningless. Fortunately, Theorem 2 shows that our AIL can help to avoid this problem by encouraging the independence between and , then it will be hard for to approximate solely by . Thus, will have to select useful features from . The proof can be found in Appendix A.3.
3.4. Extension Considering Prior Knowledge
As described in Section 3.2, the feature attribution layer (the output layer of the explainer) is in the middle of networks. Since it is optimized by an end-to-end learning process, the information transmission is inefficient. Therefore, we propose to involve efficient interpretations as good priors of feature attribution.
Let be a feature-importance-score vector generated by another interpretation method for a sample and the model . Assume for , and , which can be easily achieved through a softmax operation.
Given for a sample , we can regard as for , where denotes whether the th feature should be selected. Similarly, we can regard as . Then assuming conditional independence between the interpretation models and given and , we can obtain
The derivation details can be found in Appendix A.4.
Nonetheless, as we expect that the end-to-end learning process can generate better explanations, the prior explanation should decrease its influence when the number of epochs increases. Therefore, we define
In addition, we add a constraint for the explanations, learning to be close to for a loss :
The constraint will fade out when the number of epochs becomes large, and thus only contributes for a warm start.
We conduct comprehensive evaluations on five datasets:
Fashion-MNIST dataset (Xiao et al., 2017) to classify Pullover and Coat,
our established mobile sensor dataset from Tencent Honor of Kings game for teenager recognition, which we refer to as Tencent Gaming Dataset (TGD) in this paper.
The detailed re-organization process for each data will be introduced in the following sections.
4.0.1. Methods for Comparison
We compare our method (Ours) with the state-of-the-art model-agnostic baselines: LIME (Ribeiro et al., 2016), kernel SHAP (SHAP) (Lundberg and Lee, 2017), CXPlain (CXP) (Schwab and Karlen, 2019), INFD (Yeh et al., 2019), L2X (Chen et al., 2018) and VIBI (Bang et al., 2019). We also compare model-specific baselines: Gradient (Grad) (Simonyan et al., 2013) and Gradient Input (GI) (Shrikumar et al., 2017).
4.0.2. Evaluation Metrics
We follow Chen (Chen et al., 2018) to perform a predictive evaluation for the fidelity of both the selected and the unselected features. For the Fidelity of the Selected features, given an explanation, e.g., selected features , from an arbitrary IFS interpretation method, we evaluate whether the given model truly use the selected features primarily to generate the very output . To answer this, we need to approximate based on selected features . Thus, we evaluate by the consistency between and (recall that is with unselected features imputed by zeros), denoted by FS-M(%). However, is trained on all features of , not on (Hooker et al., 2019). Therefore, we additionally propose to evaluate the consistency between and as a reference, denoted by FS-A(%), where is an trained approximator on to learn the mapping . High FS-M or FS-A result suggests high importance of the selected features. Similarly, for the Fidelity of the Unselected features, we evaluate the consistency between and , denoted by FU-M(%); and the consistency between and , denoted by FU-A(%), where is an trained approximator on to learn the mapping . Low FU-M or FU-A result suggests high importance of the selected features. Note that low FS-A or high FU-A is possible because the number of selected features are usually small. Nonetheless, simultaneous results of high FS-A and low FU-A suggest good selected features.
For human evaluation, we also follow Chen (Chen et al., 2018) to evaluate Fidelities of Selected features denoted by FS-H(%), i.e., whether the predictions made by a human using selected features are consistent with those made by the given model using all the features. We adopt this metric to evaluate whether human can understand how the given model makes decisions. Note that sometimes human may not understand or be satisfied with features selected by the given model. After all, we are explaining the given model, not human.
For the evaluation metric for the fidelities, we report top-1 accuracy (ACC@1), since the five tasks are all binary classification. Specifically, the model outputs are transformed to categorical variables to compute accuracy. We adopt binary masks to select features,
For the evaluation metric for the fidelities, we report top-1 accuracy (ACC@1), since the five tasks are all binary classification. Specifically, the model outputs are transformed to categorical variables to compute accuracy. We adopt binary masks to select features,i.e., top values of are set to , others are set to , and then we treat as the selected features. On the other hand, we evaluate the influence of adversarial examples on the feature importance scores by the sensitivity score, SEN(%), proposed by Yeh et al. (Yeh et al., 2019). We also report the average explanation Time (by second) Per Sample (TPS) on a single NVidia Tesla m40 GPU.
4.0.3. Implementation Details
In Eq. (6), adopts cross-entropy. adopts cross-entropy for IMDB, MNIST, and TGD, and adopts Wasserstein distance (Lee et al., 2019) for Fashion-MNIST and ImageNet. The weights for in Eq. (7) and (8) are 1e-3 for MNIST and 1 for all other datasets. We adopt the GI method to provide the prior explanations. in Eq. (13) is mean absolute error with the weight to be 1e-3 for ImageNet and 0 for others.
for all the datasets. We constrain the model-capacity of each method to be the same to acquire fair comparisons. For each dataset, we split half test data for validation. For each method on each dataset, we repeat 20 times and report the averaged results. Our implementation uses Keras with Tensorflow(Abadi et al., 2016) backends. We list all other details in Appendix B.
The IMDB (Maas et al., 2011) is a sentiment analysis dataset which consists of 50,000 movie reviews labeled by positive/negative sentiments. Half reviews are for training, and the other half is for testing. We split half of the testing reviews for validation. For the given model, we follow Chen et al. (Chen et al., 2018)
to train a 1D convolutional neural network (CNN) for binary sentiment classification, and achieve the test accuracy of 88.60%. We develop our approximator with the same architecture as the given model. And we develop our explainer with the 1D CNN used by L2X(Chen et al., 2018) with a global and a local component. For a fair comparison, each method selects top- important words as an explanation for a review.
As shown in Table 1, our method significantly outperforms state-of-the-art baseline methods. Especially, our FS-M score shows nearly optimal fidelity, which is objectively validated by the original given model. Given that our FU-A score is similar to those of baselines, which shows that our selected features are indeed important, we demonstrate that the effectiveness and superiority of our method are significant. We present some examples of selected words of our method and the state-of-the-art baseline VIBI in Fig. 2. As shown in Fig. 2, our method not only selects more accurate keywords, but also provides more interpretable word combinations, such as “I enjoyed it”, “fabulous dancer”, “extremely bad movie”, and “excellent thriller”. Even though “I”, “it”, “dancer”, “movie”, and “thriller” are not apparent whether they are positive or negative words. Especially in Fig. 2 (2), our method picks the word “Oscar”, which is not explicit positive, but its underlying logic suggests positive sentiment. These inspiring examples support the significant superiority of our method.
4.1.1. Human Evaluation
We also evaluate with the help of humans to quantify how interpretable are those selected words. We randomly select reviews in the testing set for this human evaluation. We invite Tencent employees to infer the sentiment of a review given only the selected words. The explanations provided by different interpretation methods are randomly mixed before sent to these employees. The final prediction of each review is averaged over multiple human annotations. For the explanations that are difficult to infer the sentiments, we ask the employees to provide random guesses. Finally, as shown by the FS-H scores in Table 1, our method significantly outperforms baseline methods as well.
4.1.2. Ablation Study
We evaluate three variants of our method by ablating our three components, i.e., the model output feedback (Output), AIL, and prior knowledge-based warm start (Prior). In Table 2, we show the effectiveness of both the model output feedback and AIL. It is worthy of mentioning that, although the warm start strategy does not improve the final scores, it boosts the convergence rate at the start of optimization, as shown in Fig. 3.
4.1.3. Sanity Check
We perform the model and data randomization tests suggested by Adebayo et al. (Adebayo et al., 2018b). We evaluate the sanity by the cosine correlation between binary masks, i.e., the original mask, and the other one resulted from randomization. The sanity scores for model and data randomization tests are 9.39% and 10.25%, respectively, which shows that our explanations are dependent on models and then are valid.
We select the data of 3 and 8 of the MNIST dataset (LeCun et al., 1998) for binary classification with 11, 982 training and 1984 testing images. We train a 2D CNN with two convolutional layers for the classification and achieve the test accuracy of 99.89%. We develop our approximators are by the same architecture as the given model, whereas we develop our explainer by a 2D CNN with three convolutional layers only. Each method selects top- pixels as an explanation for an image.
As shown in Table 3, our method still outperforms state-of-the-art model-agnostic baseline methods except for the CXPlain method, which uses the additional true label for each sample and is highly sensitive to adversarial examples. The Grad and GI methods are model-specific and not robust when facing challenging data, e.g., see Tables 1, 4, and 5. Compared with the next-best model-agnostic baseline VIBI, the strength of our method determined by the FU-M score is to select the necessary features of a sample for recognition. We show some examples of selected pixels of our method and VIBI in Fig. 4. As shown in Fig. 4 (1), the VIBI masked image is closer to 8 than 3, whereas our masked image is more similar to 3. In Fig. 4 (2), though the VIBI masked image is similar to 8, but it is also close to a 3. In contrast, our masked image can never be 3. Since we are interpreting the recognition logic of model rather than that of humans, it is important to select features in favor of the machine logic, e.g., considering both possibility and impossibility.
4.2.1. Human Evaluation
We randomly select images in the testing set for this human evaluation. We invite Tencent employees who are experts in machine learning to perform the same binary classification given only the masked images. Specifically, we ask the subjects to provide two scores in the range of for each image. Each score is for the possibility of each class (3 or 8). We perform
normalization for the scores as the final prediction probabilities. Other settings and procedures are similar to Section4.1.1. Finally, as shown by the FS-H scores in Table 3, our method significantly outperforms baseline methods as well.
The Fashion-MNIST dataset (Xiao et al., 2017) is a dataset of Zalando’s article images, which consists of images of classes. We select the data of Pullover and Shirt for binary classification dataset with 12,000 training and 2000 testing images. We train a 2D CNN with the same architecture as that for MNIST for the classification and achieve the test accuracy of 92.20%. The architectures of our approximators and explainer are also the same as those for MNIST. Each method selects top- pixels as an explanation for an image.
As shown in Table 4, our method outperforms state-of-the-art baseline methods except for the INFD method. Since INFD adds perturbation to each feature and directly performs regression, it is suitable for well-aligned data, e.g., the Fashion-MNIST dataset. However, as shown in Tables 1 and 3, INFD’s performances are disappointing for data that are not well-aligned. Therefore, our method is more robust. Moreover, INFD is extremely time-consuming to be applied to practical applications and is sensitive to adversarial examples. Compared with the next-best baseline VIBI, we show some examples of selected pixels in Fig. 5. As shown in Fig. 5, VIBI primarily focuses on the contours, whereas our method focuses on relatively fixed local regions. Since the data are well-aligned, the explanations provided by our method are more consistent with the machine logic of the original model.
We select the data of Gorilla and Zebra from ImageNet (Deng et al., 2009) for binary classification. We adopt the MobileNet (Howard et al., 2017) and train only the top layer for the classification and achieve the test accuracy of 100%. We develop our approximators with the same architecture and adopt the U-Net (Ronneberger et al., 2015) for our explainer. Each method selects top-10% pixels as an explanation for an image. As shown in Table 5, our method outperforms state-of-the-art baselines. We exhibit some examples of selected pixels in Fig. 6 and compare them with the best baseline VIBI. As shown in Fig. 6 (1), our selected pixels are more concentrated on the label-related regions, which demonstrates that our method can improve the model identifiability. Fig. 6 (2) shows that our method can better avoid irrelevant regions, e.g., the ground and the back of an ostrich.
Last, we apply our method to the Tencent Gaming Dataset (TGD), which consists of 100 million samples from 5 million gamers. Each sample is a time-series data with the time dimension and feature dimension of and
, respectively. We extract the features from inertia sensors and touch information of mobile phones, in both time and frequency domains, and categorize ingroups. Each feature vector of a sample corresponds to a -second operation during the game. Three vectors are ordered by time, but not necessarily continuous in time. The learning task is the teenage gamer (age ) recognition. The original model is a stacked LSTM with an accuracy of 90.16%. The approximator uses the same structure, and the explainer is also a stacked LSTM. Our method achieves the FS-M, FU-M, FS-A, FU-A, and SEN scores of 95.68%, 82.24%, 95.33%, 82.37%, and 0.18%, respectively, selecting only 10% of features. We show examples of selected features in Fig. 7. In Fig. 7 (1), the teenage gamer performs a complex operation excitedly at the start but performs a monotonous/regular operation at the end. Whereas, in Fig. 7 (2), the adult gamer starts with casual flipping of the mobile phone, and ends with a complex/skilled operation.
In this paper, we investigate the model interpretation problem in the favored direction of Instance-wise Feature Selection (IFS). We propose a Model-agnostic Effective Efficient Direct (MEED) IFS framework for model interpretation. Specifically, we consider the model output feedback as an additional input to learn an explainer to mitigate the sanity and information transmission problems. Furthermore, we propose an adversarial infidelity learning (AIL) mechanism to screen relative unimportant features for mitigating the combinatorial shortcuts and the model identifiability problems. Our theoretical analyses show that AIL can mitigate the model identifiability and combinatorial shortcuts problems. Our experimental results reveal that AIL can mitigate the model identifiability problem and learn more necessary features. Moreover, our extension to integrate efficient interpretation methods as proper priors has been shown to provide a warm start and mitigate the information transmission problem. Comprehensive empirical evaluation results provided by quantitative metrics and human evaluation demonstrate the effectiveness, superiority, and robustness of our proposed method across benchmarks and one real-world Tencent game dataset.
- Tensorflow: a system for large-scale machine learning.. In OSDI, Vol. 16, pp. 265–283. Cited by: §4.0.3.
- Local explanation methods for deep neural networks lack sensitivity to parameter values. arXiv preprint arXiv:1810.03307. Cited by: §2.
- Sanity checks for saliency maps. In Advances in Neural Information Processing Systems, pp. 9505–9515. Cited by: 4th item, 1st item, §4.1.3.
A unified view of gradient-based attribution methods for deep neural networks.
NIPS 2017-Workshop on Interpreting, Explaining and Visualizing Deep Learning, Cited by: §1, footnote 1.
- Survey and critique of techniques for extracting rules from trained artificial neural networks. Knowledge-based systems 8 (6), pp. 373–389. Cited by: §2.
- On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one 10 (7), pp. e0130140. Cited by: §2.
- Explaining a black-box using deep variational information bottleneck approach. arXiv preprint arXiv:1902.06918. Cited by: 2nd item, §1, §2, §3.2, §4.0.1.
- Explicability? legibility? predictability? transparency? privacy? security? the emerging landscape of interpretable agent behavior. In Proceedings of the International Conference on Automated Planning and Scheduling, Vol. 29, pp. 86–96. Cited by: §1.
- Learning to explain: an information-theoretic perspective on model interpretation. In International Conference on Machine Learning, pp. 883–892. Cited by: §A.1, §B.1, 2nd item, §1, §1, §1, §1, §2, §3.2, §3.2, §4.0.1, §4.0.2, §4.0.2, §4.1.
- Imagenet: a large-scale hierarchical image database. In , pp. 248–255. Cited by: 4th item, §4.4.
- Explanations can be manipulated and geometry is to blame. In Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), pp. 13567–13578. External Links: Cited by: 3rd item.
- Techniques for interpretable machine learning. Communications of the ACM 63 (1), pp. 68–77. Cited by: §1.
- Feature attribution as feature selection. External Links: Cited by: 2nd item.
- Fooling neural network interpretations via adversarial model manipulation. In Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), pp. 2921–2932. External Links: Cited by: 3rd item.
- A benchmark for interpretability methods in deep neural networks. In Advances in Neural Information Processing Systems, pp. 9734–9745. Cited by: 2nd item, §1, §3.2, §4.0.2.
- Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861. Cited by: §4.4.
- Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 3543–3556. Cited by: 2nd item.
- The relativistic discriminator: a key element missing from standard gan. arXiv preprint arXiv:1807.00734. Cited by: §B.1.
- Explaining neural networks via perturbing important learned features. arXiv preprint arXiv:1911.11081. Cited by: 2nd item.
- Gradient-based learning applied to document recognition. Proceedings of the IEEE 86 (11), pp. 2278–2324. Cited by: 2nd item, §4.2.
- Sliced wasserstein discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10285–10295. Cited by: §B.1, §4.0.3.
- The mythos of model interpretability. Queue 16 (3), pp. 31–57. Cited by: §1.
- A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, pp. 4765–4774. Cited by: §2, §4.0.1.
- Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies-volume 1, pp. 142–150. Cited by: 1st item, §4.1.
- Fast and efficient information transmission with burst spikes in deep spiking neural networks. In 2019 56th ACM/IEEE Design Automation Conference (DAC), pp. 1–6. Cited by: 4th item.
- Model agnostic supervised local explanations. In Advances in Neural Information Processing Systems, pp. 2515–2524. Cited by: footnote 1.
- Why should i trust you?: explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135–1144. Cited by: 1st item, 2nd item, §2, §4.0.1.
- U-net: convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pp. 234–241. Cited by: §B.1, §4.4.
Capturing the essence: towards the automated generation of transparent behavior models.
Eleventh Artificial Intelligence and Interactive Digital Entertainment Conference, Cited by: §2.
- CXPlain: causal explanations for model interpretation under uncertainty. In Advances in Neural Information Processing Systems, pp. 10220–10230. Cited by: 2nd item, §1, §2, §4.0.1.
- Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 3145–3153. Cited by: §2, §4.0.1.
- Deep inside convolutional networks: visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034. Cited by: §2, §4.0.1.
- Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825. Cited by: §2.
- Striving for simplicity: the all convolutional net. In ICLR (workshop track), Cited by: §2.
- Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 3319–3328. Cited by: §1, §2.
- Should health care demand interpretable artificial intelligence or accept “black box” medicine?. Annals of Internal Medicine. Cited by: §1.
Attention is not not explanation.
Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 11–20. Cited by: 2nd item.
- Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747. Cited by: 3rd item, §4.3.
- On the (in) fidelity and sensitivity of explanations. In Advances in Neural Information Processing Systems, pp. 10965–10976. Cited by: 2nd item, 3rd item, §1, §2, §3, §4.0.1, §4.0.2, footnote 1.
- Interpretable deep learning under fire. arXiv preprint arXiv:1812.00891. Cited by: 3rd item.
Adversarial attention modeling for multi-dimensional emotion regression. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 471–480. Cited by: §2.
Appendix A Proofs and Derivations
a.1. Proof of Theorem 1
The proof follows that of Theorem 1 of Chen et al. (Chen et al., 2018).
(1) Forward direction: Given the definition of , we have for any pair , and any explainer ,
In the case when is a set instead of a singleton, we identify with any distribution that assigns arbitrary probability to each elements in , and with zero probability outside . With abuse of notation, indicates both the set function that maps every pair to a set and any real-valued function that maps to an element in . Taking expectation over the distribution of , and adding at both sides, we have
for any explainer .
(2) Reverse direction: The reverse direction is proved by contradiction. Since the optimal explanation satisfies
for any other , assume the optimal explanation is such that there exists a set of nonzero probability, over which does not degenerates to an element in . Concretely, we define as
For any , we have
where is a deterministic function in the set of distributions that assign arbitrary probability to each elements in , and with zero probability outside . Outside , we always have
which is a contradiction to Eq. (16). ∎
a.2. Derivations for The Variational Lower Bounds
First, for selected features, we have:
For any , we obtain the lower bound by applying the Jensen’s inequality:
It is similar for unselected features. ∎
a.3. Proof of Theorem 2
Therefore, our AIL mechanism learns to minimize . Since
the mutual information is minimized. By the property of mutual information, we assume that the minimization of encourages the independence between and , which leads to
Thus, by marginalizing at both sides, we have
Therefore, the independence between and is encouraged as well. Since and are deterministic between each other, then for any set such that , there exists a fixed set such that , and vice versa. Thus we have for any set ,
Thus, the independence between and is also encouraged. ∎
a.4. Derivation for Eq. (11)
The third equation is from the assumption of conditional independence between the interpretation models and given and . Assuming for all , because we have no knowledge of the explainer, then we have
Appendix B Implementation Details
b.1. Details for Our Method
For with cross entropy loss, in Eq. (8) we still minimize , and just replace the target for by , following the suggestion of the Relativistic GAN et al. (Jolicoeur-Martineau, 2018). For with the Sliced Wasserstein distance (Lee et al., 2019), the number of random vectors is for Fashion-MNIST and for ImageNet.
For IMDB, we adopt the structure of the original model of L2X (Chen et al., 2018) for the same structure for our original model and approximators. We adopt adopt the structure of the explainer of L2X (Chen et al., 2018) for the structure for our explainer, with a global and a local component. For MNIST and Fashion-MNIST, the structure of our original model and approximators is shown in Table 6. Whereas the structure for our explainer is shown in Table 7
. For ImageNet, we adopt the MobileNet module in the package of keras.applications.mobilenet without the top layer as our explainer. The model parameters pretrained on ImageNet are fixed. We only stack a global max-pooling layer and learn a full-connected top layer. We adopt the preprocess_input function in the keras.applications.mobilenet package for image pre-processing. We perform a max-pooling of kernel size and stride of, before generate the feature important scores, and perform an up-sampling with kernel size and stride of when masking pixels. We adopt the adopt the U-Net (Ronneberger et al., 2015) for our explainer, whose structure is complex and then omitted due to space limitations. Similarly, the structures of our modules for TGD are also omitted. Readers may refer to the publicly-available code for more implementation details.
For IMDB, the model output is input to a MLP with three hidden layers with neurons and the ReLu activation, before being concatenated to the global component of the explainer. For image datasets, the model output is linearly mapped to the same shape with the first channel of an image and is concatenated to the raw image as an additional channel. For TGD, the model output is linearly mapped to the same shape as an raw data sample and concatenated to the data sample in the feature dimension.
|Layer||# Filters||Kernel Size||Stride||
|Layer||# Filters||Kernel Size||Stride||# Padding||Activation|
b.2. Details for Baseline Methods
For Grad, we compute the gradient of the selected class with respect to the input feature and uses the absolute values as importance scores. We perform summation operations to form the importance scores with proper shapes. For GI, the gradient is multiplied by the input feature before calculate the absolute value. For INFD, we select the Noisy Baseline method for consistent comparisons, since its another method Square is only suitable for image datasets. The structures of explainers are the same for CXP, L2X, VIBI, and Ours. The structures of original models and approximators are the same for L2X, VIBI, and Ours.
The hyper-parameters of each method are tuned according the strategy mentioned in their respective papers.
On ImageNet, for all the baseline methods, we perform a max-pooling of kernel size and stride of for the feature important scores, and perform an up-sampling with kernel size and stride of when masking pixels.