Ontology-based Interpretable Machine Learning for Textual Data

In this paper, we introduce a novel interpreting framework that learns an interpretable model based on an ontology-based sampling technique to explain agnostic prediction models. Different from existing approaches, our algorithm considers contextual correlation among words, described in domain knowledge ontologies, to generate semantic explanations. To narrow down the search space for explanations, which is a major problem of long and complicated text data, we design a learnable anchor algorithm, to better extract explanations locally. A set of regulations is further introduced, regarding combining learned interpretable representations with anchors to generate comprehensible semantic explanations. An extensive experiment conducted on two real-world datasets shows that our approach generates more precise and insightful explanations compared with baseline approaches.



There are no comments yet.


page 1

page 2

page 3

page 4


Locally Interpretable Model Agnostic Explanations using Gaussian Processes

Owing to tremendous performance improvements in data-intensive domains, ...

Interpretable Active Learning

Active learning has long been a topic of study in machine learning. Howe...

Semantic XAI for contextualized demand forecasting explanations

The paper proposes a novel architecture for explainable AI based on sema...

Model-Agnostic Explainability for Visual Search

What makes two images similar? We propose new approaches to generate mod...

Nothing Else Matters: Model-Agnostic Explanations By Identifying Prediction Invariance

At the core of interpretable machine learning is the question of whether...

On the Art and Science of Machine Learning Explanations

This text discusses several explanatory methods that go beyond the error...

Interpretable Data-Based Explanations for Fairness Debugging

A wide variety of fairness metrics and eXplainable Artificial Intelligen...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

In critical scenarios, such as clinical practices, having the ability to interpret machine learning (ML) model outcomes is significant to reduce the error rate and improve the trustworthiness of ML-based systems [27, 12]. To achieve this, typical approaches, called Interpretable ML (IML), are to train additional interpretable models to generate explanations, which usually are crucial features (i.e., important terms, in text analysis [20, 25] or super-pixels, in image processing [9, 29]), for each predicted outcome. However, most of existing IML algorithms usually treat input features independently, without considering their semantic correlations, especially in natural language processing. As a result, generated explanations commonly are fragmented, incomplete, and difficult to understand.

Addressing this problem is a non-trivial task, since: (1) It is difficult to capture semantic correlations among features, which can be contextually rich and dynamic; (2) There is still a lack of scientific study on how to integrate semantic correlations among features into IML to generate semantic explanations, which are concise, complete, and easy to understand; and (3) The search space for semantic explanations can be large and complicated, given noisy and poor data. That results in a limited understanding of how to define semantic explanations, and effectively and efficiently identify them.

In literature, ontology, which encodes domain knowledge, can be used to capture semantic correlations among input features, such as entities, terms, phrases, concepts, etc. [23, 6]. However, there is an unexplored gap regarding how to guide the learning process of an IML model based on ontology. Straightforwardly matching ontology and explaining data points, by randomly sampling co-occurring terms and concepts in conventional approaches, e.g., LIME [25], may not generate semantic explanations, since contextual information in the data is usually rich and complicated compared with the ontology. In addition, building an ontology that can sufficiently capture contextual information in the data is costly. Meanwhile, the traditional concept of anchor texts [26] can be used to narrow down the search space, by pinpointing generally important texts. However, the approach was not designed for each single and independent data point, i.e., at local level.

Our contributions. To synergistically overcome these challenging issues, we propose a novel Ontology-based IML (OnML

) to generate semantic explanations, by intergrating domain knowledge encoded in ontology and information extraction techniques into IML. In this paper, we consider a text classification model, in which text data is classified into different categories. Then, we learn a linear interpretable model by approximating the predictive model based on data sampled around the prediction outcome.

In order to achieve our goals, we first present a new concept of ontology-based tuples, each of which essentially is a set of correlated terms, words, and concepts semantically encoded and co-existed in the ontology and textual data. Departing from existing approaches, we identify and integrate ontology-based tuples into a new sampling approach, in which the semantic correlations among terms, words, and concepts are sampled and captured, instead of utilizing each of them independently.

Second, we propose a new concept of learnable anchor texts, to narrow down the search space for explanations. A learnable anchor text essentially is a contextual phrase, which can be expanded by adding nearby terms. For instance, anchors can be started with a predefined seed term having negative meanings, e.g., “no,” “not,” “illegal,” and then be expanded to neighboring texts in order to effectively capture negative experiences and events, e.g., “not get any help.” Anchors, which have the highest importance scores measuring their impacts upon the model outcome, will be chosen.

Third, we introduce a set of regulations to combine ontology-based tuples, anchor texts, and triplexes extracted from the text, to generate semantic explanations. Each explanation is assigned an importance score. To our knowledge, OnML establishes the first connection among domain knowledge ontology, IML, and learnable anchor texts. Such a mechanism will greatly extend the applicability of ML, by fortifying the models in both interpretability and trustworthiness.

Finally, extensive experiments conducted on two real-word datasets in critical applications, including drug abuse in the Twitter-sphere [14] and consumer complaint analysis111https://www.consumerfinance.gov/data-research/consumer-complaints/, to quantitatively and qualitatively evaluate our OnML, show that our algorithm generates concise, complete, and easy-to-understand explanations, compared with existing mechanisms.

Ii Background and Problem Definition

In this section, we revisit IML, ontology-based approaches, and information extraction algorithms, which are often used to generate explanations. We further discuss the relation to previous frameworks and introduce our problem definition.

Let be a database that consists of samples, each of which is a sample associated with its label . Each

is a one-hot vector of

categories . A classifier outputs class scores that maps an input to a vector of scores s.t. and

. The highest-score class is selected as the predicted label for the sample. By minimizing a loss function

that penalizes a mismatching between the prediction and the original value , an optimal classifier is selected.

Interpretable Machine Learning. Let us briefly revisit IML, starting with the definition of interpretable model. Given an interpretable model , which provides insights and qualitative understanding about the prediction model given an input , there are two important criteria in learning : 1) local fidelity, which implies the ability of interpretable models to approximate the prediction model in a vicinity of the input, and 2) interpretability, which is the sufficiently low complexity of interpretable models that make humans easy to understand the explanations. In textual data, the complexity, denoted as , usually is the number of important words [20, 25], based upon that users can easily handle to evaluate the generated explanations.

Let be a sample of , where is generated by randomly selecting or removing features/words in . is a similarity function to measure the proximity between and . Given a -dimensional binary vector , indicates that the feature -th is present in , and vice-versa.

To achieve the interpretability and local fidelity, Ribeiro et al. [25] minimize a loss function , with a low complexity , by solving the following problem:


where , is an exponential kernel with is a distance function (e.g., cosine distance for textual data) with a width , and .

To obtain the data for learning in Eq. 1, sampling approaches are employed. In LIME [25], the authors draw nonzero elements of the original data uniformly at random. Similar to this approach, a number of works follow [22, 1, 15]. Apart from the randomization, model decomposition is another line of learning [27, 20], in which the prediction is decomposed on individual features to learn the effect of each feature on the outcome. These existing randomization and decomposition approaches treat features independently; therefore, they cannot capture correlations among features. This may not be practical in real-world scenarios, since features usually are highly and semantically correlated.

Ontology-based Approaches. To capture semantic correlations among input features and ontology can be applied. Ontology is used in [11] to filter and rank concepts from selected data points to conduct informative explanations. The explanations are derived in ontological forms. For example, the information, “a 30 year-old individual, with an operation occurred in 1989,” can be conveyed by the representation, “TheSilentGeneration OperationIn1980s.” (TheSilentGeneration denotes people in the age range of 30-39.) However, building a rich contextual ontology is expensive, so typically ontology only captures a limited number of core concepts and their correlations. This is the reason why ontological forms cannot capture all common sense knowledge in the textual information. In reality, humans generally use natural languages in a variety of text presentations. Therefore, an appropriate combination of a single-form ontology with other approaches to generate semantic explanations is necessary.

Fig. 1: A flow chart of the OnML approach.
Fig. 2: Drug abuse ontology.

In [6]

, Confalonieri et al. use ontology to learn an understandable decision tree, which is an approximation of a neural network classifier. Explanations are in a non-syntactic form, and they are not designed to explain a single and independent data point. Different from

[6], we aim at generating semantic explanations for each input . In this paper, generating semantic explanations is defined as a process of mapping a text to a representation of important information in a syntactic and understandable form.

Information Extraction. Apart from IML, information extraction (IE) is another direction to capture contextual information semantically. The first Open IE algorithm is TextRunner [4]

, which identifies arbitrary relation phrases in English sentences by automatically labeling data using heuristics for training the extractor. Following

[4], a number of Open IE [34, 31, 8] were introduced. Unfortunately, these approaches ignore the context. OLLIE [28] includes contextual information; and extracts relations mediated by nouns, adjectives, and verbs; and outputs triplexes (subject, predicate, object). Compared to Open IE approaches, our algorithm mainly focuses on generating semantic explanations associated with the prediction label.

Iii Ontology-based Interpretable Machine Learning for Textual Data

In this section, we formally present our proposed OnML framework (Fig. 1). Alg. 1 presents the main steps of our approach. Given an input , an ontology , and a set of all concepts in , we first present the notion of ontology-based tuples (Line 3), which will be used in an ontology-based sampling technique to learn the interpretable model (Lines 4-6). Next, we learn potential anchor texts using the input and the model (Line 7). Meanwhile, OLIIE [28] is applied to extract triplexes, which have high confident scores, in (Line 8). After learning , learning anchor texts , and extracting triplexes , we introduce a set of regulations to combine them together to generate semantic explanations (Line 9). Let us first present the notion of ontology-based tuples as follows.

Iii-a Ontology-based Tuples

Given concepts and , is used to indicate that has a directed connection to . In considerably correlated domains, such as text data, it is observed that 1) words appeared near to each other in a sentence have the same contextual information, and 2) different sentences usually have different contextual information. To encode the observations, we introduce a contextual constraint, as follows:


where and are two words in , is a predefined threshold, and measures the distance between the positions of and in . In text data, if and belong to two sentences, they are considered to be violating the contextual constraint. Intuitively, the constraint is used to connect words 1) that appear near to each other in a sentence (contextual correlated) and 2) that belong to connected concepts in the ontology (conceptual correlated). If there is no contextual constraint, there can be mismatched information between the domain knowledge and the explanation extracted in the text.

Definition 1.

Ontology-based tuple. Given and in , is called an ontology-based tuple, if and only if: (1) s.t. and ; (2) ; and (3) .

Since ontology has directed connections among its concepts, ontology-based tuples are asymmetric, i.e., and are different. For the sake of clarity without affecting the generality of the approach, we use a drug abuse ontology as an example (Fig. 2). Given the drug abuse ontology and as “She uses orange juice and does not like weed. She knows that smoke causes addiction and headache.”, list of ontology-based words can be found {use, weed, smoke, addiction, headache}. These words are in “Abuse Behavior” (use and smoke), “Drug” (weed), “Side Effect” (addiction), and “Symptom” (headache) concepts. Following the aforementioned conditions (Eq. 2 with ), two ontology-based tuples are found, which are (smoke, addiction) and (smoke, headache). In the meantime, (addiction, headache) and (weed, smoke) are not ontology-based tuples, since there is no directed connection between the “Side Effect” concept and the “Symptom” concept, and “weed” and “smoke” are in different sentences. By using the contextual constraint, we can eliminate “use weed,” which is contextually incorrect, from the explanation.

1:  Input: Input ; ontology , and user-predefined anchor
2:  Classify by a prediction model
3:  Find ontology-based tuples in based on concepts and relations in
4:  Sample , based on ontology-based tuples found by our sampling technique to obtain sampled data
5:  Generate vectors of predictive scores with
6:  Learn an interpretable model based on and by Eq. 1
7:  Learn anchor text by our anchor learning algorithm (Alg. 2)
8:  Extract triplexes in using an existing Open IE technique
9:  Combine ontology-based tuples, learned anchors, and extracted triplexes by our proposed regulations
10:  Output: Semantic explanation
Algorithm 1 OnML approach

Iii-B Ontology-based Sampling Technique

To integrate ontology-based tuples into learning , we introduce a novel ontology-based sampling technique. To learn the local behavior of in its vicinity (Eq. 1), we approximate by drawing samples based on , with the proximity indicated by . A sample can be sampled as:


where and

are probabilities randomly drawn for each word

and words together, respectively. If is greater than a predefined threshold, then the word(s) will be included in .

In our sampling process, and , i.e., an ontology-based tuple, are sampled together as a single element. This aims to integrate the semantic correlation between and , captured in an ontology-based tuple into the sampling process. In fact, we are sampling the semantic correlation, but not sampling each word/feature or independently. This enables us to measure the impact of this semantic correlation on . In addition, words, which are not in any ontology-based tuple, are sampled independently. After sampling (Eq. 3), we obtain the dataset that consists of sampled data points associated with its label . is used to learn by solving Eq. 1.

Iii-C Learnable Anchor Text

Before presenting our anchor mechanism, we introduce an importance score notion, which will be used to choose the best anchor and calculate the importance of generated explanations.

Iii-C1 Importance Score

To get insights into the importance of generated explanations and their impacts upon the model outcome, we calculate an importance score () for each explanation. Intuitively, the higher importance score, the more important the explanation is. is calculated as:


where is the original text excluding words in the explanation and is average coefficients of associated with all words in .

1:  Input: Input ; prediction model ; number of sentences in , denoted as ; user-predefined anchors
2:   set of anchors for )
3:  for  do
4:     if any appears in the sentence  then
5:         Denote as a set of ordered words appearing after in the sentence in
6:          ( is a set of candidate anchors)
7:          ( is a set of importance scores, associated with each candidate anchor)
8:         for  do
9:            ; ;
10:          Choose the best anchor for sentence :
11:     else
14:  Output:
Algorithm 2 Anchors learning algorithm

Iii-C2 Anchor Text Learning Mechanism

It is challenging to work with long and poor data, e.g., large number of words, or misspelled text, since the contextual information is generally rich and complicated. Building an ontology to adequately represent such data is expensive, and insufficient in many cases. That results in a large undercovered search space for explanations. To address this problem, we introduce a learnable anchor mechanism to narrow down the search space.

The learning anchor technique is presented in Alg. 2. The anchor is initialized with an empty set (Line 2). A set of user-predefined anchors is provided, which consists of starting-words that are further expanded by incrementally adding words to the end of the sentence. Then, the importance score of each candidate anchor is calculated, following Eq. 4. The top-1 anchor , which has the highest important score, for each sentence are then chosen.

Iii-D Generating Semantic Explanations

We further apply OLLIE [28] to extract triplexes (subject, predicate, and object) to identify the syntactic structure in a sentence, which can shape our explanations in a readable form. To generate semantic explanations , we introduce a set of regulations to combine , , and together:
1) with is a set of all words in .
2) If there is no ontology-based tuple found, will only consist of the learned anchor texts.
3) In a sentence, if there are two or more ontology-based tuples, we introduce four rules to merge them together:

  • Simplification:

    • Given and , if and are in the same concept, then the ontology-based explanation is .

    • Given and , if and are in the same concept, then the ontology-based explanation is .

    • Given and , then the ontology-based explanation is .

  • Union: Given , , , and , the ontology-based explanation is .

  • Adding Causal words: Semantic explanation can be in the form of a causal relation. Thus, if a causal word, e.g., “because,” “since,” “therefore,” “while,” “whereas,” “thus,” “thereby,” “meanwhile, “however,” “hence,” “otherwise,” “consequently,” “when,” “whenever” appears between any words in ontology-based tuples/explanations, we add the word to the explanation, following its position in .

  • Combining with anchor texts and triplexes : After having ontology-based explanations, we combine them with and based on their positions in . Then, the semantic explanation is generated from the beginning towards the end of all positions of words found in the ontology-based explanations, , and . For example, in the sentences, “We were filling out all the forms in the application. However, there is a letter in saying loss mitigation application denied for not sending information to us.”, after the learning process, we obtain: 1) ontology-based explanation is (loss, application); 2) anchor text is “not sending information;” and 3) triple is “a letter; denied; mitigation application.” The explanation is “a letter in saying loss mitigation application denied for not sending information.”

4) If different ontology-based tuples are in different sentences in , due to the contextual constraint in Eq. 2, the explanation for each sentence follows the 3 regulation.

It is worthy noting that we use aforementioned regulations to combine ontology-based tuples to be a longer ontological term. This makes the ontology used in a much better representation rather than independent and direct connections .

Fig. 3: Visualization of drug abuse (top) and consumer complaint (bottom) experiments.

Iv Experiment

We have conducted extensive experiments on two real-world datasets, including drug abuse (Twitter-sphere [14]) and consumer complaint analysis from Consumer Financial Protection Bureau.

Iv-a Baseline Approaches

Our OnML approach is evaluated in comparison with traditional approaches: (1) an interpretable model-agnostic explanation, i.e., LIME [25]; and (2) information extraction, i.e., OLLIE [28]. LIME is one of the state-of-the-art and well-applied approaches in IML, in which the predictions of any model are explained in a local region near the sample being explained. There are other algorithms sharing the same spirit as LIME, in terms of generating explanations [18, 29, 30, 33, 32, 3]. For the sake of clarity, we use LIME as a representative baseline regarding this line of research.

The key differences among OnML, OLLIE, and LIME are that OnML leverages domain knowledge to tie the explanations up to the predicted label and considers correlations among words in textual data to generate semantic explanations. Meanwhile, OLLIE focuses more on grammatical analysis to extract triples from the text. LIME generates fragmented interpretable components by learning a linear interpretable model locally around the prediction outcome and weight these components using coefficients of the interpretable model. In LIME and OLLIE, domain knowledge is not used.

Iv-B Datasets and Domain Ontologies

To validate the proposed method, we have developed two different domain ontologies, which are drug abuse ontology (Fig. 2) and consumer complaint ontology (in the Appendix) These ontologies were constructed for certain domains (e.g., drug abuse and consumer complaint) since it is necessary to capture specific semantic and causal relations among components. As default in Protégé [17], each arrow represented by its color demonstrates a certain type of causal relation in which its tail represents a domain and its head represents a range of the relation. For example, in the drug abuse ontology (Fig.2

), purple arrow is for “is involved with” with domain is “Drug” and range is “Abuse Behavior” while green arrows are for “suffer from”. These ontologies were semi-manually generated, in which concepts were grouped and collected from the dataset by K-means clustering algorithm

[10], and then judged by humans to reduce inappropriate concepts.

Iv-B1 Drug Abuse Dataset

We will use the term “drug abuse” in the wider sense, including abuse and use of Schedule 1 drugs that are illegal and have no medical use (e.g. legal painkiller and weed) or illegally (e.g. getting drugs without prescription or even from blackmarket); and misuse of Schedule 2 drugs, which have medical uses, yet have a potential for severe addiction, and which can be life-threatening [5]. The drug abuse ontology captures different concepts collected from drug abuse tweets, grouped by K-means clustering algorithm, and then finalized by our team experts. Main concepts of the drug abuse ontology (DrugAO) (Fig. 2) capture correlation among key concepts, including abuse behaviors, drug types, drug sources, drug users, symptoms, side effects, and medical condition when using drug. Abuse behaviors concept is about behaviors of abusers, such as abuse, addict, blunt, etc. Drug types consists of different types of legal and illegal drugs, e.g., narcotics, cocaine, and weed. Drug sources is where drug users, who are the main objects of the ontology, gets drugs from. Symptoms and side effects are about different negative short-term and long-term effects of drugs on users. Medical condition contains terms about expression of disease and illness caused by using drugs. In total, DrugAO has drug-abuse related terms (including slang terms and street names), and relations.

The drug abuse dataset (Table I) consists of tweets labelled by [14] with a high agreement score. Among them, tweets are drug abuse tweets, labeled positive and the rest are non drug abuse tweets, labeled negative.

Iv-B2 Consumer Complaint Dataset

A consumer complaint is defined, here, as a complaint about a range of consumer financial products and services, sent to companies for response. In complaints, consumers typically talk about their mortgage-related issues, such as: (1) Applying for a mortgage or refinancing an existing mortgage (application, credit decision, underwriting); (2) Closing on a mortgage (closing process, confusing or missing disclosures, cost); (3) Trouble during payment process (loan servicing, payment processing, escrow accounts); (4) Struggling to pay mortgage (loan modification, behind on payments, foreclosure); (5) Problem with credit report or credit score; (6) Problem with fraud alerts or security freezes, credit monitoring or identity theft protection services; and (7) Incorrect information on consumer’s report or improper use of consumer’s report. Main concepts of the consumer complaint ontology (ConsO) (Fig. 4) encode the relation among different entities related to consumer complaint: for instance, who is complaining; what happened to make consumers unhappy and then complaint; etc. There are six major concepts in ConsO, which are thing in role, complaint, event, event outcome, property, and product. Thing in role is people and organizations related to complaint, such as buyers, investors, dealers, et,. Event and event outcome are about negative events happened that cause consumer complaints. Property is things belonging to consumers and product is substances of some parties (e.g., banks) offering to consumers. In total, we have finance and product-related terms and relations covered in our ontology. The consumer complaint dataset consists of mortgage-related complaints, labeled with categories. These complaints were used for learning a model to predict the issue regarding each complaint.

Fig. 4: Consumer complaint ontology.

Iv-C Experimental Settings

Our experiment focuses on validating whether: (1) Our OnML approach can be applied on different agnostic predictive models; and (2) Our approach can generate better explanations, compared with baseline approaches, in both quantitative and qualitative measures. Our ontologies, code, and data are available on Github222https://github.com/PhungLai728/OnML.

To achieve our goal, we carry out our evaluation through three approaches. First, by employing SVM and LSTM, we aim to illustrate that OnML works well with different agnostic predictive models. Second, we leverage the word deleting approach [2] as an quantitative evaluation. Third, we apply qualitative evaluation with Amazon Mechanical Turk (AMT).

Iv-C1 Model Configurations

In the drug abuse dataset, tweets were vectorized by TF-IDF [24] and then classified by a linear kernel SVM model. We achieved accuracy. Tweets are short, i.e., the average and maximum numbers of words in a tweet are 12 and 37 (Table I). Therefore, it is not necessary to apply the anchor learning algorithm, which is designed to tighten down the search space for long text data.

In the consumer complaint dataset, Word2vec [21]

is applied for feature vectorization. Then, a Long short-term memory (LSTM)

[13] is trained as a prediction model. In LSTM, we used an embedding input layer with , one hidden layer of

hidden neurons, and a softmax output layer with

outputs. An efficient ADAM [16] optimization algorithm with learning rate was employed to train LSTM. For the prediction model, we achieved accuracy. We registered that this is a reliable performance, since the categories are densely correlated resulting in a lower prediction accuracy [7]. Another reason for the low accuracy is the limited number of samples. We will collect more data in the future.

For sufficiently learning anchors in consumer complaints, we have chosen a set of negative terms as user-predefined anchors not, no, illegal, against, without. Importance scores in LIME are weights of the linear interpretable model. With OLLIE, importance scores of extracted triplexes are calculated in the same way as in our method (as shown in Eq. 4). LIME and OLLIE settings are used as default in [25, 28]. We only show OLLIE rules which have the confidence score greater than and top- words from LIME. The contextual constraint in Eq. 2 is for drug abuse and for consumer complaint dataset. The pre-defined threshold in Eq. 3 is .

To be fair, we also combined the learned anchors to the results of OLLIE. In addition, another variation of our algorithm is to combine ontology-based terms and anchors, called Ontology algorithm. This is further used to comprehensively evaluate our proposed approach.

Iv-C2 Quantitative Evaluation

We use the word deleting approach [2], which deletes a sequence of words from a text and then re-classifies the text with missing words. By differences between the original text and the missing text, we examine the importance of the explanation to the prediction. Accuracy changes (AC) and prediction score changes (SC) are as:


where the higher values of AC and SC indicate the more important explanations derived.

[width=12em]StatisticsDataset Drug abuse
of samples 9,700 13,965
of categories 2 16
Max of words/sentence 37 4,893
Mean of words/sentence 12 285
TABLE I: Data statistical analysis.
Accuracy changes (%) Score changes (%)
LIME 15.04 26.98
OLLIE 15.47 23.52
OnML 25.52 33.48
TABLE II: AC and SC in drug abuse.

In our experiment, we deleted the top- highest importance score explanations in OnML and OLLIE approaches and the top- highest weighted words in LIME. To be fair, is the number of words in the -deleted explanations in OnML. In drug abuse, since the tweet is typically short, and so there are not many explanations generated. In consumer complaint classifying, .

Fig. 5: AMT experiment results.
Fig. 6: Average score changes in consumer complaint.

Iv-C3 Qualitative Evaluation

We recruit human subjects on Amazon Mechanical Turk (AMT). This is a common means of evaluation for the needs of qualitative investigation by humans [19, 29]. Detailed guidance for each experiment is provided to users before they conduct the task.

We asked AMT workers to choose the best explanation by seeing side-by-side explanation algorithms. On top of that, we provided the original tweet/ complaint associated with their labels and prediction results. The visualization showing explanation results of the approaches is in Fig. 3. It is important to note that, in our real experiment, to avoid bias, name of each algorithm is hidden, and their positions in the visualization are randomized.

We were recruiting users/tweets in the drug abuse and users/complaints in the consumer complaint experiment. To quantify the voting results from AMT users, we use: (1) Count the total number of votes, called normal count, i.e., the best algorithm is chosen over all votes ( users/complaint complaints); and (2) Count the majority number of votes, called majority count, i.e., the best algorithm for each complaint is the algorithm of the largest number over votes.

Iv-D Experimental Results and Analysis

To evaluate the interpretability of each approach, positive tweets and complaints, randomly selected, were used.

Iv-D1 Drug Abuse Explanation

As in Table II, the accuracy is deducted significantly, and the predictive score changes the most in OnML. In fact, the values of AC and SC are 25.52% and 33.48% given OnML, compared with 15.47% and 23.52% given OLIIE, and 15.04% and 26.98% given LIME. This demonstrates that the explanations generated by our algorithm are more significant, compared with the ones generated by baseline approaches. In the evaluation by humans using AMT (Fig. 3), OnML clearly outperforms LIME and OLLIE. Text in the tweet is generally short and can be represented by several key words. Therefore, individual words learned by LIME can be sufficient to generate more insightful explanations. Meanwhile, OLLIE tends to extract all possible triplexes in the text, which can be redundant and wordy explanations.

Iv-D2 Consumer Complaint Explanation

The results on the consumer complain dataset further strengthen our results. Fig. 6 shows SC after deleting top-, top-, and top- explanations from OnML, Ontology, and OLLIE, as well as after deleting the most important words in LIME. In all three cases, score changes in OnML have the highest values, indicating that the explanations generated by OnML are the most significant to the prediction. In the evaluation by humans using AMT (Fig. 3), our OnML algorithm outperforms baseline approaches. Ontology approach achieves higher results than LIME and OLLIE. This shows the effectiveness of the ontology-based approach. LIME does not consider semantic correlations among words, resulting in a poor outcome.

Iv-D3 Completeness and Concision

In Fig. 3 (top), OnML generates “i smoking weed,” which provides concise and complete information about why it is predicted as a drug abuse tweet (smoking weed) and who was doing it (i) in a syntactic form. Meanwhile, 1) LIME derives relevant words to drug abuse (i.e., weed, smoking) without considering the correlation among these words; and 2) OLLIE generates lengthy and somewhat irrelevant explanations, e.g., “chinese food; be eating on; a roof.” In Fig. 3 (bottom), OnML derived semantic explanations for consumer complaints, which tell us that consumers were facing issues in loan refinance, e.g., “called fha and they clain that fha does not review loans.” Compared to OnML, Ontology generates laconic explanations, e.g., “fha loan” that give no sense of why consumer complaints. LIME provides a set of fragmented words and OLLIE generates wordy explanations, which are difficult to follow. More examples of explanation results are in the Appendix.

Our key observations are: (1) Combining ontology-based tuples, learnable anchor texts, and information extraction can generate complete, concise, and insightful explanations to interpret the prediction model ; and (2) Our OnML model outperforms other baseline approaches in both the quantitative and qualitative experiments, showing a promising result.

V Conclusion

In this paper, we proposed a novel ontology-based IML to generate semantic explanations, by integrating interpretable models, ontologies, and information extraction techniques. A new ontology-based sampling technique was introduced, to encode semantic correlations among features/terms in learning interpretable representations. An anchor learning algorithm was designed to limit the search space of semantic explanations. Then, a set of regulations for connecting learned ontology-based tuples, anchor texts, and extracted triplexes is introduced, to produce semantic explanations. Our approach achieves a better performance, in terms of semantic explanations, compared with baseline approaches, illustrating a better interpretability into ML models and data. Our approach paves an early brick on a new road towards gaining insights into machine learning using domain knowledge.

Acknowledgment. The authors gratefully acknowledge the support from the National Science Foundation (NSF) grants CNS-1747798, CNS-1850094, and NSF Center for Big Learning (Oregon). We thank for the financial support from Wells Fargo. We also thank Nisansa de Silva (University of Oregon, USA) for valuable discussions.


  • [1] A. Adhikari, D. M. Tax, R. Satta, and Ma. Fath (2018) Example and feature importance-based explanations for black-box machine learning models. arXiv preprint arXiv:1812.09044. Cited by: §II.
  • [2] L. Arras, F. Horn, G. Montavon, K. R. Müller, and W. Samek (2017) “What is relevant in a text document?”: an interpretable machine learning approach. PloS one 12 (8), pp. e0181142. Cited by: §IV-C2, §IV-C.
  • [3] S. Bach, A. Binder, G. Montavon, F. Klauschen, K. R. Müller, and W. Samek (2015) On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one 10 (7), pp. e0130140. Cited by: §IV-A.
  • [4] M. Banko, M. J. Cafarella, S. Soderland, M. Broadhead, and O. Etzioni (2007) Open information extraction from the web.. In IJCAI, Vol. 7, pp. 2670–2676. Cited by: §II.
  • [5] S. Barlas (2013) Prescription drug abuse hits hospitals hard: tighter federal steps aim to deflate crisis. Pharmacy and Therapeutics 38 (9), pp. 531. Cited by: §IV-B1.
  • [6] R. Confalonieri, F. M. delPrado, S. Agramunt, D. Malagarriga, D. Faggion, T. Weyde, and T. R. Besold (2019) An ontology-based approach to explaining artificial neural networks. arXiv preprint arXiv:1906.08362. Cited by: §I, §II.
  • [7] J. Deng, A. C. Berg, K. Li, and L. Fei-Fei (2010) What does classifying more than 10,000 image categories tell us?. In ECCV, pp. 71–84. Cited by: §IV-C1.
  • [8] A. Fader, S. Soderland, and O. Etzioni (2011) Identifying relations for open information extraction. In EMNLP, pp. 1535–1545. Cited by: §II.
  • [9] R. C. Fong and A. Vedaldi (2017) Interpretable explanations of black boxes by meaningful perturbation. In ICCV, pp. 3429–3437. Cited by: §I.
  • [10] E. W. Forgy (1965) Cluster analysis of multivariate data: efficiency versus interpretability of classifications. Biometrics 21, pp. 768–769. Cited by: §IV-B.
  • [11] L. Freddy and W. Jiewen (2018) Semantic explanations of predictions. arXiv:1805.10587v1. External Links: Link Cited by: §II.
  • [12] Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh (2017) Making the V in VQA matter: elevating the role of image understanding in visual question answering. In CVPR, pp. 6904–6913. Cited by: §I.
  • [13] S. Hochreiter and J. Schmidhuber (1997) Long short-term memory. Neural computation 9 (8), pp. 1735–1780. Cited by: §IV-C1.
  • [14] H. Hu, N. Phan, J. Geller, S. Iezzi, H. Vo, D. Dou, and S. A. Chun (2019)

    An ensemble deep learning model for drug abuse detection in sparse twitter-sphere

    In MEDINFO’19), Cited by: §I, §IV-B1, §IV.
  • [15] Y. Jia, J. Bailey, K. Ramamohanarao, C. Leckie, and M. E. Houle (2019) Improving the quality of explanations with local embedding perturbations. Cited by: §II.
  • [16] D. P. Kingma and J. Ba (2014) ADAM: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §IV-C1.
  • [17] H. Knublauch, R. W. Fergerson, N. F. Noy, and M. A. Musen (2004) The protégé owl plugin: an open development environment for semantic web applications. In ISWC, pp. 229–243. Cited by: §IV-B.
  • [18] S. M. Lundberg and S.-I. Lee (2017) A unified approach to interpreting model predictions. In NeurIPS, pp. 4765–4774. Cited by: §IV-A.
  • [19] D. Martens, B. Baesens, T. Van G., and J. Vanthienen (2007)

    Comprehensible credit scoring models using rule extraction from support vector machines

    EJOR 183 (3), pp. 1466–1476. Cited by: §IV-C3.
  • [20] D. Martens and F. Provost (2013) Explaining data-driven document classifications. Cited by: §I, §II, §II.
  • [21] T. Mikolov, K. Chen, G. S. Corrado, and J. A. Dean (2015) Computing numeric representations of words in a high-dimensional space. Google Patents. Note: US Patent 9,037,464 Cited by: §IV-C1.
  • [22] S. Nagrecha, J. Z. Dillon, and N. V. Chawla (2017) MOOC dropout prediction: lessons learned from making pipelines interpretable. In WWW, pp. 351–359. Cited by: §II.
  • [23] H. Phan, D. Dou, H. Wang, D. Kil, and B. Piniewski (2017) Ontology-based deep learning for human behavior prediction with explanations in health social networks. Information Sciences 384, pp. 298–313. Cited by: §I.
  • [24] J. Ramos et al. (2003) Using TF-IDF to determine word relevance in document queries. In iCML, Vol. 242, pp. 133–142. Cited by: §IV-C1.
  • [25] M. T. Ribeiro, S. Singh, and C. Guestrin (2016) Why should i trust you?: explaining the predictions of any classifier. In KDD, pp. 1135–1144. Cited by: §I, §I, §II, §II, §II, §IV-A, §IV-C1.
  • [26] M. T. Ribeiro, S. Singh, and C. Guestrin (2018) Anchors: high-precision model-agnostic explanations. In AAAI, Cited by: §I.
  • [27] S. M. Robnik and I. Kononenko (2008) Explaining classifications for individual instances. TKDE 20 (5), pp. 589–600. Cited by: §I, §II.
  • [28] M. Schmitz, R. Bart, S. Soderland, O. Etzioni, et al. (2012) Open language learning for information extraction. In EMNLP-IJCNLP, pp. 523–534. Cited by: §II, §III-D, §III, §IV-A, §IV-C1.
  • [29] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra (2017) Grad-cam: visual explanations from deep networks via gradient-based localization. In ICCV, pp. 618–626. Cited by: §I, §IV-A, §IV-C3.
  • [30] A. Shrikumar, P. Greenside, and A. Kundaje (2017) Learning important features through propagating activation differences. In ICML, pp. 3145–3153. Cited by: §IV-A.
  • [31] S. Soderland, B. Roof, B. Qin, S. Xu, O. Etzioni, et al. (2010) Adapting open information extraction to domain-specific relations. AI magazine 31 (3), pp. 93–102. Cited by: §II.
  • [32] J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller (2014) Striving for simplicity: the all convolutional net. arXiv preprint arXiv:1412.6806. Cited by: §IV-A.
  • [33] M. Sundararajan, A. Taly, and Q. Yan (2016) Gradients of counterfactuals. arXiv preprint arXiv:1611.02639. Cited by: §IV-A.
  • [34] F. Wu and D. S. Weld (2010) Open information extraction using wikipedia. In ACL, pp. 118–127. Cited by: §II.