Joint Abductive and Inductive Neural Logical Reasoning

Neural logical reasoning (NLR) is a fundamental task in knowledge discovery and artificial intelligence. NLR aims at answering multi-hop queries with logical operations on structured knowledge bases based on distributed representations of queries and answers. While previous neural logical reasoners can give specific entity-level answers, i.e., perform inductive reasoning from the perspective of logic theory, they are not able to provide descriptive concept-level answers, i.e., perform abductive reasoning, where each concept is a summary of a set of entities. In particular, the abductive reasoning task attempts to infer the explanations of each query with descriptive concepts, which make answers comprehensible to users and is of great usefulness in the field of applied ontology. In this work, we formulate the problem of the joint abductive and inductive neural logical reasoning (AI-NLR), solving which needs to address challenges in incorporating, representing, and operating on concepts. We propose an original solution named ABIN for AI-NLR. Firstly, we incorporate description logic-based ontological axioms to provide the source of concepts. Then, we represent concepts and queries as fuzzy sets, i.e., sets whose elements have degrees of membership, to bridge concepts and queries with entities. Moreover, we design operators involving concepts on top of the fuzzy set representation of concepts and queries for optimization and inference. Extensive experimental results on two real-world datasets demonstrate the effectiveness of ABIN for AI-NLR.

READ FULL TEXT VIEW PDF

page 1

page 2

page 3

page 4

05/28/2022

TFLEX: Temporal Feature-Logic Embedding Framework for Complex Reasoning over Temporal Knowledge Graph

Multi-hop logical reasoning over knowledge graph (KG) plays a fundamenta...
08/05/2021

Fuzzy Logic based Logical Query Answering on Knowledge Graph

Answering complex First-Order Logical (FOL) queries on large-scale incom...
05/07/2020

Détermination Automatique des Fonctions d'Appartenance et Interrogation Flexible et Coopérative des Bases de Données

Flexible querying of DB allows to extend DBMS in order to support imprec...
06/26/2021

A Neural-symbolic Approach for Ontology-mediated Query Answering

Recently, low-dimensional vector space representations of knowledge grap...
06/16/2019

Self-organized inductive reasoning with NeMuS

Neural Multi-Space (NeMuS) is a weighted multi-space representation for ...
09/29/2021

Holland’s Method as a Material in Environmental Education

Abstract: In this work, the Holland’s methodology is modified for the ca...
05/01/2021

A Peek Into the Reasoning of Neural Networks: Interpreting with Structural Visual Concepts

Despite substantial progress in applying neural networks (NN) to a wide ...

1. Introduction

Figure 1. An example of Abductive and Inductive Neural Logical Reasoning. The query is “who will be interested in something that G. Hinton is investigating and Google is deploying?”. The answers are not only extensional entity-level answers represented by yellow circles: Meta, Amazon, MIT, and Y. LeCun, but also intensional concept-level answers represented in the squares: AI Researchers, The Academia, and The Industry.

Along with the rapid development of high-quality large-scale knowledge infrastructures (Tanon et al., 2020; Auer et al., 2007)

, researchers are increasingly interested in exploiting knowledge bases for real-world applications, such as knowledge graph completion

(Bordes et al., 2013) and entity alignment (Trisedya et al., 2019). However, to take advantage of knowledge bases, a fundamental yet challenging task still remains unsolved, i.e., neural logical reasoning (NLR), which attempts to answer complex structured queries that include logical operations and multi-hop projections given the facts in knowledge bases with distributed representations (Hamilton et al., 2018). Recently, efforts (Hamilton et al., 2018; Ren et al., 2019; Ren and Leskovec, 2020) have been made to develop NLR systems by designing strategies to learn geometric or uncertainty-aware distributed query representations, and proposing mechanisms to deal with various logical operations on these distributed representations.

However, existing neural logical reasoners cannot fully fulfill our needs. In many real-world scenarios, we not only expect specific entity-level answers, but also seek for more descriptive concept-level answers, where each of the concepts is a summary of a set of entities. For example, as shown in Figure 1, the query asks “who will be interested in something that G. Hinton is investigating and Google is deploying?”, users expect entity-level answers like Meta, Amazon, MIT, and Y. LeCun, as well as concept-level answers such as AI Researchers, The Academia, and The Industry. In this example, the conceptual answer The Academia refers to a summary of a set consisting of Y. LeCun and MIT, and it is intuitively desirable for users and worth exploring. In biomedical applications, people may want to find the causes for a set of symptoms and expect both entity-level answers (such as SARS-CoV-2 causing Fever) as well as concept-level answers (such as Viral infections causing Fever). In this case, the answer constitutes a descriptive concept-level answer (e.g., Viral infections) that is a summary of a set of entity-level answers. Downstream tasks like online chatbots (Liu et al., 2018) and conversational recommender systems (Zhou et al., 2020) also need to retrieve rich and comprehensive answers to provide better services. Thus, providing both entity-level and concept-level answers can highly improve their capability of generating more informative responses to users and enriching the semantic information in answers for downstream tasks.

Figure 2. The considered types of queries represented with their graphical structures. and represent the intersection and union logical operations, respectively. Squares denote concepts and circles represent entities.

From the perspective of logic theory, the ability of providing both concept-level and entity-level answers corresponds to the capability of jointly supporting abductive and inductive neural logical reasoning (AI-NLR). That is to say, previous NLR systems support inductive reasoning, while more general AI-NLR systems additionally support abductive reasoning which is highly useful in the context of query answering over ontologies (Elsenbroich et al., 2006). Specifically, neural inductive reasoners seek to infer extensional answers by listing the elements of the set of answers, while neural abductive reasoners aim at inferring explanations from given observations in the form of concepts, and therefore are providing intensional answers. In other words, neural inductive reasoning infers the set of entities that satisfies each query, while neural abductive reasoning infers explanations of each query by summarizing the entity-level answers with descriptive concepts. Therefore, the AI-NLR problem is more general than the regular NLR problem in terms of including not only induction that gives extensional entity-level answers, but also abduction that yields intensional concept-level answers as explanations. Note that such descriptive concepts are higher-level abstractions of the set of entities and are more informative than the set in some cases (Cook, 2009).

From the perspective of users and downstream tasks, AI-NLR is helpful by jointly providing the more informative concept-level and entity-level answers. From the perspective of logic theory, AI-NLR is useful in ontological applications by jointly supporting abductive and inductive reasoning. However, existing methods (Hamilton et al., 2018; Ren et al., 2019; Ren and Leskovec, 2020) can hardly reach AI-NLR for the following reasons. On the one hand, concepts are excluded from the NLR systems. That is to say, previous NLR systems perform reasoning upon regular knowledge graphs where only entities and relations exist. On the other hand, mechanisms for exploiting concepts have not been well established. Specifically, previous solutions only measure query-entity similarity for NLR, without considering concept representations and operators involving concepts, such as query–concept similarity for neural abductive reasoning.

Along this line, we propose an original solution named ABIN for AI-NLR. ABIN is a shortname which stands for an ABduction and INduction reasoner. The key challenges for addressing AI-NLR are the incorporation of concepts, representation of concepts, and operator on concepts. First, we observe that ontologies include taxonomic hierarchies of concepts, concept definitions, and concept subsumption relations (Gruber, 1993). To incorporate concepts into the AI-NLR system, we thus introduce some description logic based ontological axioms into the system to provide sources of concepts. Second, we find that fuzzy sets (Klir and Yuan, 1995), i.e., sets whose elements have degrees of membership, can naturally bridge entities with concepts, i.e., vague sets of entities. Therefore, we represent concepts as fuzzy sets in ABIN. Meanwhile, properly representing queries is the prerequisite of effectively operating on concepts. We find that fuzzy sets can also bridge entities with queries, i.e., vague sets of entity-level answers. The theoretically-supported, vague, and unparameterized fuzzy set operations enable us to resolve logical operations within queries. Thus, the adoption of fuzzy sets is an ideal solution for concepts and queries representation in AI-NLR. Then operators involving concepts can also be designed based on fuzzy sets, including query-concept operators for abduction, entity-concept operators for instantiation, and concept-concept operator for subsumption. Attributed to the well designed operators, a joint abductive and inductive neural logical reasoner can be achieved.

We summarize the main contribution of this work as follows:

  • To the best of our knowledge, we are the first to focus on the AI-NLR problem that aims at providing both entity-level and concept-level answers so as to jointly achieve inductive and abductive neural logical reasoning, which better satisfies the need of users, downstream tasks, and ontological applications;

  • We propose an original solution ABIN that properly incorporates, represents, and operates on concepts. We incorporate ontologies to provide sources of concepts and employ fuzzy sets as the representations of concepts and queries. Logical operations are supported by the well-established fuzzy set theory and operators involving concepts are rationally designed upon fuzzy sets;

  • We conduct extensive experiments and demonstrate the effectiveness of ABIN for AI-NLR. We publish in public two pre-processed benchmark datasets for AI-NLR and the implementation of ABIN to foster further research111https://github.com/lilv98/ABIN.

2. Related Work

2.1. Neural Logical Reasoning

Given the vital role of neural logical reasoning (NLR) in knowledge discovery and artificial intelligence, great efforts have been made to develop sophisticated NLR systems in recent years. GQE (Hamilton et al., 2018) is the pioneering work in this field, the authors formulate the NLR problem and propose to simply use points in the embeddings space to represent logical queries. Q2B (Ren et al., 2019) claimed that the representation of each query in the embedding space should be a geometric region instead of a single point because each query is equivalent to a set of entity-level answers in the embedding space. Therefore, they use hyper-rectangles that can include multiple points in the embedding space to represent queries. HypE (Choudhary et al., 2021) extended Q2B by using hyperboloids in a Poincaré ball for distributed query representation. However, these methods can only deal with projection, intersection, and union, without considering negation. More recently, BetaE (Ren and Leskovec, 2020)

proposed to use Beta distributions over the embedding space to represent queries for NLR. The advantage of using distributions over points or geometric regions is that they can properly handle queries with negations and queries without answers. However, these reasoners could only give extensional entity-level answers, while we focus on the more general AI-NLR problem that aims at additionally providing descriptive concepts to summarize the entity-level answers. We bring neural logical reasoners to the stage of abductive and inductive reasoning.

2.2. Fuzzy Logic for NLR

Besides representing logical queries as points, geometric reagions, or distributions, more recent methods explore fuzzy logic (Klir and Yuan, 1995) for NLR. CQD (Arakelyan et al., 2020) used -norm and -conorms from the fuzzy logic theory to achieve high performance on zero-shot settings. More specifically, mechanisms are proposed for the inference stage on various types of queries, while only training the simple neural link predictor on triples (1p queries in Figure 2). FuzzQE (Chen et al., 2021b)

directly represented entities and queries using embeddings with specially designed restrictions and interpreted them as fuzzy sets for NLR. However, this study still focuses on the regular NLR problem, while we are solving a more general problem that additionally includes neural abductive reasoning. Furthermore, we explicitly include concepts and represent them as fuzzy sets, whereas FuzzQE represents only the query as fuzzy set. Moreover, CQD only uses fuzzy logic at the entity level and FuzzQE uses fuzzy sets with arbitrary numbers of elements as the tunable embedding dimension without reasonable interpretations. We interpret queries as fuzzy sets where each element represents the probability of an entity being an answer, aligning with the definition and the essence of fuzzy sets

(Klir and Yuan, 1995), i.e., sets where each element has a degree of membership. This allows us to fully exploit fuzzy logic and provides a theoretical foundation in fuzzy set theory for our work.

2.3. Ontology Representation Learning

Recently, several methods that exploit ontologies from the perspective of distributed representation learning have been developed (Kulmanov et al., 2021). ELEm (Kulmanov et al., 2019) and EmEL (Mondala et al., 2021) learn geometric embeddings for concepts in ontologies. The key idea of learning geometric embeddings is that the embedding function projects the symbols used to formalize axioms into an interpretation of these symbols such that is a model of the ontology. Other approaches (Smaili et al., 2019; Chen et al., 2021a) rely on regular graph embeddings or word embeddings and apply them to ontology axioms. Another line of research (Hao et al., 2019, 2020) focuses on jointly embedding entities and relations in regular knowledge graphs, as well as concepts and roles (relations) in ontological axioms. Our work is related to ontology representation learning in that we incorporate some description logic based ontological axioms in Section 3.1.2 to provide sources of concepts, and we exploit concepts with distributed representation learning in our proposed ABIN for AI-NLR. Methods for representation learning with ontologies have previously only been used to answer link prediction tasks such as predicting protein–protein interactions or performing knowledge graph completion, which can be viewed as answering 1p queries in Figure 2 whereas we also focus on more complex queries as well as providing abductive concept-level answers.

3. Methodology

Incorporating, representing, and operating on concepts are the three key components for an abductive and inductive neural logical reasoner. In this section, we first formulate the AI-NLR problem along with the process of incorporating concepts into the reasoning system. Then we propose an original solution ABIN for AI-NLR by designing concept representations and operators involving concepts. We introduce optimization and inference procedures in the end.

Figure 3. Illustration of the description logic based ontological knowledge base .

3.1. Incorporating Concepts

3.1.1. Regular NLR

The regular NLR problem is defined on knowledge graphs. A knowledge graph is formulated as , where , , denote the head entity, relation, and tail entity in triple , respectively, and refer to the entity set and the relation set in .

In the context of NLR, as shown in Figure 2.(a), each triple is regarded as a positive sample of the 1p query with an answer that satisfies , where is the anchor entity and is the projection operation with relation . Furthermore, the regular NLR problem may also address the intersection, union, and negation operations , , and within queries. Thus, infinite types of queries can be found with the combinations of these logical operations. We consider the representative types of queries, which are listed and demonstrated with their graphical structures in Figure 2. For example, queries of type pi in Figure 2.(c) are to ask .

Regular neural logical reasoners seek to provide extensional entity-level answers for each query. In particular, the answers are a set of entities that satisfies the query by inductive reasoning. Following the definition of inductive reasoning (Vickers, 2009), the given knowledge graph might give us very good reason to accept that each element of the set is an answer of the query, but it does not ensure that. Therefore, we predict the possibility of each candidate entity satisfying a query . We then rank the possibilities and select the top- entities in as the set of answers. Since all the candidate answers are entities, we can only retrieve extensional entity-level answers from the regular NLR systems.

3.1.2. Abductive and Inductive NLR

A joint abductive and inductive neural logical reasoner is upon ontological axioms provided in a knowledge base

, which is an ordered pair (

, ) for TBox and ABox , where is a finite set of terminological axioms and is a finite set of assertion axioms. Specifically, terminological axioms within a TBox are of the form where the symbol denotes subsumption (). In general, and can be concept descriptions that consist of concept names, quantifiers and roles (relations), and logical operators; we limit ABIN to axioms where and are concept names that will not involve roles or logical operators (Baader, 2003). In the followings, we do not distinguish between a concept name and a concept description unless there are special needs. Then, a TBox is:

(1)

where denotes the set of concept names in . accounts for the source of concepts and the pairwise concept subsumption information in the AI-NLR system. Assertion axioms in consist of two parts. The one part is the role assertion that is expressed as:

(2)

where denote entities, denotes the entity set in , denotes the role assertion between and , and is the the role set of . accounts for the triple-wise relational information about entities and roles in AI-NLR. The other part within is the concept instantiation between an entity and an concept :

(3)

where represents is an instance of . serves as the bridge between and by providing pairwise links between entities and concepts.

Since we incorporate concepts in the AI-NLR systems, we are able to ask questions about concepts. In particular, for a query of arbitrary type discussed in Section 3.1.1, we not only perform inductive reasoning that returns a set of entities of as the extensional entity-level answers, but also we perform abductive reasoning that infers an explanation for each query result by summarizing entity-level answers with descriptive concepts, yielding another set of concept-level answers as the intensional concept-level answers. More specifically, as shown in Figure 2, the answers are no longer restricted to be (represented as circles), they can also be (represented as squares). To achieve this goal, we predict the possibility of each candidate entity as well as the possibility of each candidate concept satisfying a query . We then rank predicted scores of candidate entities and predicted scores of candidate concepts. We select and combine the top- results from each set of candidates as the final answers of with extensional entity-level and intensional concept-level answers .

Note that the regular NLR problem is a sub-problem of the AI-NLR problem. On the one hand, regular NLR systems can only provide a subset of the answers provided by abductive and inductive neural logical reasoners, i.e., . On the other hand, the entire in the context of regular NLR is equivalent to in the case of the AI-NLR problem, which is a subset of the ontological knowledge base, i.e., , leaving and with conceptual information in the ontologies not explored. Therefore, the problem we investigate is more general in terms of providing more answers and reasoning over more complex knowledge bases.

3.2. Representing Concepts and Queries

In this subsection, we first introduce how to represent concepts as fuzzy sets in our proposed ABIN for AI-NLR. Then we represent queries as fuzzy sets as well to prepare for the later operations that involve concepts and queries.

Figure 4. Illustration of the interpretation of a concept .

3.2.1. Representing Concepts

We are motivated to represent concepts as fuzzy sets by the relationship between concepts and entities. We gain insights on such relationship from the basic definition of semantics in description logics (Baader et al., 2008):

Definition 1 ().

A terminological interpretation over a signature consists of:

  • a non-empty set called the domain

  • an interpretation function that maps:

    • every entity to an element

    • every concept to a subset of

    • every role (relation) to a subset of

As we use a function-free language (Baader, 2003), we set to be the Herbrand universe (Lee, 1972) of our knowledge base, i.e., . Therefore, according to Definition 1, the interpretation of concept is a subset of , which is finite. On the other hand, fuzzy sets (Klir and Yuan, 1995) over the Herbrand Universe are finite sets whose elements have degrees of membership:

(4)

where is the membership function that measures the degree of membership of each element. Therefore, we further interpret all concepts as fuzzy sets over the finite domain as the elements of fuzzy sets . Thus, we have:

(5)

As the Herbrand universe for our language is always finite, the interpretation of concept is fully determined by the fuzzy membership function that assigns a degree of membership to each entity for , where and are the interpretation of the entity (name) set and the concept (name) set.

To obtain the degree of membership of entity in , i.e., , we first randomly initialize the embedding matrix of concepts and entities as and with Xavier uniform initialization (Glorot and Bengio, 2010), where is the embedding dimension. Then we obtain the embedding of each concept by looking up the rows of . The embedding then serves as the generator of the fuzzy set representation of each concept . Thus, we compute the similarities between each concept and every entity in our universe as the degrees of membership of each entity in the fuzzy set:

(6)

where symbol denotes matrix multiplication and represents the matrix transposition. The measured similarities are then normalized to

using the bit-wise sigmoid function

. Here, the set-wise operation to obtain consists of pair-wise operations on the entity–concept pairs; we use the same operator for Instantiation, which we will explain in Section 3.3.4.

3.2.2. Representing Queries

Properly representing queries is the prerequisite of operating on concepts. Fuzzy sets are particularly suitable to represent not only concepts, but also queries, because interpretations of queries are essentially interpretations of concepts. More accurately, queries correspond to concept descriptions that may include concept names, roles (relations), quantifiers, and logical operations. We can use the same formalism designed for representing concepts to represent entities, i.e., as a special type of fuzzy sets (Rihoux and De Meur, 2009) that assigns the membership function to for exactly one entity and to to all others. Consequently, we can interpret entities as concepts. As explained in section 3.1.1, queries may consist of entities, relations, and logical operations. Therefore, queries are interpreted as concept descriptions and we regard entities within queries as singleton concepts. Thus, we can use the same description logic semantics (Baader et al., 2017) to interpret a query and concept in Definition 1: an interpretation function maps every query to a subset of . As the Herbrand universe is finite, the interpretation of query is fully determined by the fuzzy membership function

(7)

Besides, representing queries as fuzzy sets have other advantages. Firstly, fuzzy logic theory (Klir and Yuan, 1995) well-equipped us to interpret logical operations within queries as the vague and unparameterized fuzzy set operations. The preservation of vagueness is important in that we are performing inductive and abductive reasoning that requires uncertainty, rather than deductive reasoning that guarantees the correctness. Unparameterized operations are desirable because they require fewer data during training and are often more interpretable. Secondly, since concepts are already represented as fuzzy sets, it would be more convenient for us to employ the same form of representation and retain only one form of representation within the AI-NLR system. We explain how to represent queries as fuzzy sets in detail as the followings.

Representing Atomic Queries

Each multi-hop logical query consists of one or more Atomic Queries (AQ), where an AQ is defined as a query that only contains projection(s) from an anchor entity without logical operations such as intersection , union , and negation . Therefore, the first step to represent queries is to represent AQs. We obtain the embeddings of each entity and the relation by looking up the rows of the randomly initialized entity embedding matrices and with Xavier uniform initialization (Glorot and Bengio, 2010). Then, the generator for fuzzy set representation of an valid AQ is . Thus, we obtain the fuzzy set corresponding to the query as:

(8)

Similar to the process of obtaining fuzzy set representations of concepts, Eq.(8) is to aquire the degrees of membership of every candidate being an answer to a given AQ by computing their normalized similarities.

Figure 5. Illustration of the process of representing concepts and queries for abductive reasoning.
Fusing Atomic Queries

AQs are fused by logical operations to form multi-hop logical queries. Since AQs are already represented in fuzzy sets and we are equipped with the theoretically supported fuzzy set operations, we interpret logical operations as fuzzy set operations over concepts to fuse AQs into the final query representations.

For two fuzzy sets in domain : and , we have the intersection over the two fuzzy sets as:

(9)

the union over the two fuzzy sets as:

(10)

and we have the negation over as:

(11)

where a -norm is a generalisation of conjunction in logic (Klement et al., 2004). Some examples of -norms include the Gödel -norm , the product norm , and the Łukasiewicz -norm (van Krieken et al., 2020). Analogously, a -conorm is dual to -norm and generalizes logical disjunction – given a -norm , the complementary -conorm is defined by (Arakelyan et al., 2020). The choice of the

-norm is a hyperparameter of ABIN.

Thus, each query can be decomposed into AQs and represented as a fuzzy set with Eq.(8), and then fuzzy set representations of AQs are fused by the fuzzy set operations in Eq.(9), (10), and (11) to obtain the final representation of the query. Note that fuzzy set operations hold the property of closure, which means the input and output of these operations remain fuzzy sets. Thus, the final representation of each query is also a fuzzy set .

3.3. Operating on Concepts

In previous sections, we manage to prepare for designing operators involving concepts by representing concepts and queries in fuzzy sets. Here, we design operators involving concepts for abduction, induction, subsumption, and instantiation.

3.3.1. Abduction

Abductive reasoning is to give explanations for a set of observations; here, the answers to a query are the observations that we summarize by descriptive concepts, i.e., provide as discussed in Section 3.1.2. We measure the possibility of each being an intensional concept-level answer of a given query upon fuzzy set representations. More specifically, we measure the similarity between and based on the Jensen-Shannon divergence (Endres and Schindelin, 2003)

, which is a symmetrized and smoothed version of the Kullback-Leibler divergence

. The similarity function we use for abductive inference is defined by:

(12)

where , and represent the normalized fuzzy set representations of the considered query and concept descriptions, which are given by:

(13)

where is a small value to avoid division by zero and is the exponent value in the norm formulation . are then used for model training and abductive inference in Section 3.4.

3.3.2. Induction

Inductive reasoning aims to provide extensional entity-level answers; for this purpose, only query–entity similarities need to be measured without the necessity of designing new mechanisms. Therefore, we follow the pioneering work (Hamilton et al., 2018) on NLR and to represent each query as an embedding and measure query–entity similarity for inductive reasoning:

(14)

where is the margin, denotes the function to obtain query embedding , and denotes the parameters of . We explain in detail in Section A.1.

Partition Abd-1p Abd-other Ind-1p Ind-other Sub Ins NLR-1p NLR-other
YAGO4 32,465 8,382 75 #Train 189,338 10,000 101,417 10,000 16,644 83,291 184,708 10,000
#Valid/#Test 1,000/1,000 1,000/1,000 1,000/1,000 1,000/1,000 - - 1,000/1,000 1,000/1,000
Dbpedia 28,824 981 327 #Train 473,924 10,000 136,821 10,000 2,582 225,436 362,257 10,000
#Valid/#Test 1,000/1,000 1,000/1,000 1,000/1,000 1,000/1,000 - - 1,000/1,000 1,000/1,000
Table 1. Abd, Ind, Sub, Ins, and NLR correspond to statistics of the instances for Abduction, Induction, Subsumption, Induction, and baseline NLR methods, repectively. For other query types in Figure 2, the statistics are the same for each of them.

3.3.3. Subsumption

As defined by Eq.(1), supplies for relational information among concepts with the form of concept subsumptions. Although concepts are represented in fuzzy sets and we already designed mechanism to measure the similarity between two fuzzy sets, we can not directly apply the method in Section 3.3.1

for concept subsumptions. It is because we need to measure the degree of inclusion of one concept to another instead of the similarities between them. The degree of inclusion is asymmetrical and more complex than the similarity measurement. Therefore, we employ a neural network

to model the degree of inclusion:

(15)

where symbol denotes matrix concatenation over the last dimension, and denotes the parameters of . In this paper, is a two-layer feed-forward network with activation. Note that we directly use the embeddings of concepts without interpreting concept in the Herbrand universe of entities because neither concept-entity relationships need to be modeled nor logical operations need to be resolved.

3.3.4. Instantiation

As defined by Eq.(3), bridges and by providing links between entities and concepts. Such links instantiate concept with its describing entities and thus offer relational information with the form of concept instantiation. Recall that in Section 3.2.1, we obtain the fuzzy set representation of concepts by computing the similarities between the given and every candidate with Eq.(6). In the case of concept instatiation, the set-wise computation Eq.(6) is degraded to pair-wise similarity measurement for each concept-entity pair:

(16)

where and are the categorical embeddings of concept and entity , respectively.

3.4. Optimization

Figure 6. Illustration of obtaining for optimization.

The parameters to optimize in our model ABIN include the entity embedding matrix , the concept embedding matrix for the basic representation of concepts that is out of domain , the relation embedding matrix , in Section 3.3.3, and in Section A.1. In the training stage, we sample negative samples for each positive instance of abductive reasoning by corrupting with randomly sampled (). Similarly, negative samples for induction are obtained by corrupting in with randomly sampled . For subsumption and instantiation, both sides of the concept-concept pairs and concept-entity pairs are randomly corrupted following the same procedure.

The loss of ABIN is defined as

(17)

where denotes the set of the four included task discussed Section 3.3, (or ) denotes the predicted similarity or degree of inclusion of the positive (or negative) sample according to task . The overall optimization process of is outlined in Algorithm 1 in supplementary materials.

In the inference stage, we predict (or ) for every candidate concept (or entity ) regarding to query and select the top- results to be the intensional concept-level answers (or extensional entity-level answers ) for query . Thus, we are able to achieve AI-NLR by providing the comprehensive answers . Although subsumption in Section 3.3.3 and instantiation in Section 3.3.4 are not included in the inference stage, they empowered ABIN to better represent and operate concepts by providing training instances and extra supervision signals.

MRR Hit@3
3-22 1p 2p 3p 2i 3i pi ip 2u up avg 1p 2p 3p 2i 3i pi ip 2u up avg
YAGO4 GQE (Hamilton et al., 2018) 35.3 49.5 33.7 51.3 43.9 11.8 9.1 14.4 6.6 28.4 43.7 67.4 44.9 67.1 49.9 15.0 12.3 19.7 9.4 36.6
Q2B (Ren et al., 2019) 37.3 53.4 59.6 55.0 47.6 2.1 1.6 1.7 1.1 28.8 47.1 75.4 78.2 70.0 58.2 2.8 1.9 1.9 1.4 37.4
BetaE (Ren and Leskovec, 2020) 39.0 57.0 58.9 52.9 45.8 10.4 8.9 2.7 6.5 31.3 47.8 74.7 76.7 64.2 52.5 10.1 10.2 2.9 5.0 38.2
ABIN 51.3 76.4 82.7 55.9 53.9 51.3 48.9 54.3 45.9 57.8 60.3 88.8 88.7 66.4 65.5 60.2 57.1 59.9 52.5 66.6
DBpedia GQE (Hamilton et al., 2018) 27.1 35.5 32.5 30.5 32.0 14.0 14.7 9.8 11.8 23.1 28.7 42.9 40.0 32.5 36.0 13.0 14.4 7.2 10.7 25.0
Q2B (Ren et al., 2019) 26.4 35.7 32.6 30.4 29.9 13.5 14.4 10.3 11.2 22.7 28.2 41.4 38.0 32.7 33.5 11.7 13.0 8.4 9.3 24.0
BetaE (Ren and Leskovec, 2020) 30.4 38.7 40.0 32.9 34.2 14.8 11.4 5.6 9.0 24.1 34.1 45.4 50.8 37.2 41.2 14.6 10.9 4.0 8.0 27.4
ABIN 55.0 80.8 80.7 42.9 36.7 42.0 28.5 63.4 64.9 55.0 62.4 83.7 83.6 50.9 43.7 45.0 29.9 67.2 67.3 59.3
Table 2. Abductive reasoning performance. The best results are in boldface.

4. Experiments

In this section, we conduct extensive experiments to answer the following research questions:
RQ1 How to properly compare ABIN with methods that do not support abduction?
RQ2 How does ABIN perform for abductive reasoning?
RQ3 How does ABIN perform for inductive reasoning?
RQ4 How do the introduced subsumption and instantiation perators affect the performance of ABIN?

4.1. Experimental Settings

4.1.1. Baselines (RQ1)

The considered baseline methods are the three most established methods in NLR, namely GQE (Hamilton et al., 2018), Q2B (Ren et al., 2019), and BetaE (Ren and Leskovec, 2020), which employ points, geometric regions, and distributions to represent queries, respectively. Since the regular neural logical reasoners can only provide extensional entity-level answers with inductive reasoning, we need to come up with a way to make them give concept-level answers without explicitly trained on abduction instances, so as to be compared with our proposed ABIN on abductive reasoning.

Therefore, we introduce the One-more-hop experiment to serve as baselines for the abduction task. That is, we exploit all the information given by and simply degrade concepts to entities in the training stage. Specifically, we first augment by the transductive links provided by . Then we combine the augmented and to form the new knowledge graph . Note that part of the entities in are the degraded concepts and contains an additional relation to describe the isInstanceOf relationship between an entity and a concept. Thus, we construct training examples of various types of queries using and update model parameters following (Ren et al., 2019).

In the inference stage, two sets of candidate entities are prepared for each query. One of them is the regular entity-level candidate set for inductive reasoning, which can be ranked following the original papers (Hamilton et al., 2018; Ren et al., 2019; Ren and Leskovec, 2020). The other set contains the degraded concepts for abductive reasoning. To predict the possibility of a concept being and intensional answer of a query , we add one more projection operation with the relation , so as to construct the abductive query: . In other words, abductive reasoning is implicitly achieved by an additional hop asking the isInstanceOf upon inductive queries, i.e., the One-more-hop.

4.1.2. Datasets

We conduct experiments on two commonly-used real-world large-scale knowledge bases, namely YAGO4 and DBpedia. Specifically, we use English Wikipedia version222https://yago-knowledge.org/downloads/yago-4 of YAGO4 and 2016-10 release333http://downloads.dbpedia.org/wiki-archive/downloads-2016-10.html of DBpedia. To preprocess the dataset for the AI-NLR problem, we first filter out low-degree entities in and with the threshold 5. Then we split to two sets with the ratio 95% and 5% for training and evaluation, respectively. We use the same procedure as BetaE (Ren and Leskovec, 2020) to construct instances of logical queries from . We use all the triples in in the training set as training examples of 1p queries and randomly select certain amount of training and evaluation examples for each of the other types of queries as stated in Table 1. We then split the evaluation set of each type of queries to the validation set and the testing set. We summarize the statistics of the pre-processed datasets in Table 1, where Abd, Ind, Sub, Ins, and Base represent the instances for abduction, induction, subsumption, instantiation, and the NLR baselines, respectively. The processed datasets along with the code for pre-processing will be published in public to foster further research.

MRR Hit@3
3-22 1p 2p 3p 2i 3i pi ip 2u up avg 1p 2p 3p 2i 3i pi ip 2u up avg
YAGO4 GQE (Hamilton et al., 2018) 25.8 15.8 4.6 26.4 28.4 21.7 18.1 9.6 18.2 18.7 29.7 17.3 4.6 29.0 31.4 23.5 18.6 13.6 17.4 20.6
Q2B (Ren et al., 2019) 24.5 17.2 6.8 25.7 28.8 24.5 20.2 8.6 17.7 19.3 28.4 20.0 7.0 29.5 34.4 27.0 21.3 11.6 18.5 22.0
BetaE (Ren and Leskovec, 2020) 28.2 19.7 9.5 30.1 33.4 28.3 22.0 10.6 20.7 22.5 31.4 22.3 12.0 33.5 37.5 31.3 24.1 12.5 23.8 25.4
ABIN 34.9 23.8 15.1 50.9 61.6 31.6 35.3 13.9 21.3 32.0 39.6 27.3 18.4 56.6 70.2 34.2 39.1 14.9 23.7 36.0
DBpedia GQE (Hamilton et al., 2018) 19.6 16.1 18.2 31.4 38.2 16.8 28.5 10.1 18.0 21.9 25.2 18.9 19.8 26.5 43.6 17.2 33.3 12.6 21.1 24.2
Q2B (Ren et al., 2019) 16.3 13.7 15.4 22.6 28.5 18.1 22.8 7.4 12.9 17.5 21.3 16.5 17.7 27.5 31.9 19.7 25.7 9.1 15.4 20.7
BetaE (Ren and Leskovec, 2020) 20.2 20.1 19.3 25.4 29.7 24.2 25.2 12.3 24.2 22.3 20.9 23.2 21.7 27.5 32.9 27.5 28.4 13.0 25.1 24.5
ABIN 28.8 24.5 24.4 38.4 46.3 20.1 33.6 14.0 20.6 27.9 34.6 28.0 29.0 44.6 54.8 21.4 40.6 17.8 23.0 32.6
Table 3. Inductive reasoning performance. The best results are in boldface.

4.1.3. Implementation Details

We implement ABIN using PyTorch

444https://pytorch.org/ and conduct all the experiments on Linux server with GPUs (Nvidia RTX 3090) and CPU (Intel Xeon). In the training stage, the initial learning rate of the Adam (Kingma and Ba, 2014) optimizer, the embedding dimension , and the batch size, are tuned by grid searching within {, , }, {128, 256, 512}, and {256, 512, 1024}, respectively. We keep the number of corrupted negative samples for each positive sample , the small value , the exponent value , the margin , and the adopted type of -norm as 4, , 1, 12, and , respectively. We employ early stop with validation interval of 50 and tolerance of 3 for model training. In the test phase, following (Ren et al., 2019), we use the filtered setting and report the averaged results of Mean Reciprocal Rank (MRR) and Hits@3 over 3 independent runs.

4.2. Abductive Reasoning (RQ2)

We conduct the One-more-hop experiment as described in Section 4.1.1 to answer RQ2. As shown in Table.2

, our proposed ABIN consistently outperforms baseline methods on various evaluation metric with large margins. For the

basic queries summarized in Figure 2 that are simply projections and intersections, our proposed ABIN significantly improved the performance of abductive reasoning, especially for the multiple projection queries 1p, 2p, and 3p. For extra queries in Figure.2 that are more complex in terms of including unions or combined logical operations, we even boosted the performance exponentially. The average performance of ABIN is also significantly better than baseline methods.

The superior performance of ABIN can be explained for two reasons. First, due to the lack of abductive reasoning capabilities, GQE, Q2B, and BetaE need to do reasoning over more complicated queries. For example, baseline methods need to do reasoning over an ipp query to provide intensional concept-level answers of an ip query . Therefore, 1p queries become 2p queries for baseline methods, 2p becomes 3p, and so on. Thus, the complexity of the transformed queries limits the baseline performance. Second, explicit supervision signals for abductive reasoning are not provided by the baselines. That is to say, since the concepts are degraded as entities, regular NLR methods could not explicitly feed the empirical error on concept-level answers back to update the model parameters. It is thus understandable that the baseline methods cannot perform well on abductive reasoning to provide intensional concept-level answers, especially on extra queries that are more complicated and require supervision signals more eagerly.

4.3. Inductive Reasoning (RQ3)

We formulate the AI-NLR problem as to jointly provide intensional concept-level answers, i.e., abductive reasoning, as well as extensional entity-level answers, i.e., inductive reasoning. Although ABIN is designed for AI-NLR, it is interesting to know the performance of ABIN on inductive reasoning only (for answering RQ3). The results of inductive reasoning tasks in Table 3 show that ABIN also outperforms regular NLR methods on most types of queries on various metrics. The performance gain of ABIN is credited to its capability of representing and operating on concepts. It thus has additional information of the relationships among queries, entities, and concepts, which are helpful for inductive reasoning.

Abduction 1p 2p 3p 2i 3i pi ip 2u up avg
w/o Sub 53.3 72.4 71.5 19.0 15.8 24.1 19.3 59.9 61.7 44.1
w/o Ins 52.8 68.8 67.7 38.3 35.4 34.2 19.3 56.2 61.6 48.3
ABIN 55.0 80.8 80.7 42.9 36.7 42.0 28.5 63.4 64.9 55.0
Induction 1p 2p 3p 2i 3i pi ip 2u up avg
w/o Sub 17.7 20.1 21.0 18.0 18.8 14.9 22.7 9.1 16.0 17.6
w/o Ins 17.4 18.8 19.0 18.1 18.7 14.3 22.9 8.7 17.7 17.3
ABIN 28.8 24.5 24.4 38.4 46.3 20.1 33.6 14.0 20.6 27.9
Table 4. Ablation Study on Subsumptions and Instantiation upon DBpedia dataset. The best MRR results are in boldface.

4.4. Ablation Study (RQ4)

We conduct ablation study on DBpedia dataset to answer RQ4. As shown in Table 4, when the Subsumption task is not included, i.e., ontological axioms in are not used and is not computed, ABIN w/o Sub underperforms ABIN on all types of queries for both abductive and inductive reasoning. Such results clearly demonstrate the importance of the relational information between concepts to be used for AI-NLR, and the effectiveness of the designed operator in Section 3.3.3 for handling such information. On the other hand, ABIN consistently outperforms w/o Ins on all types of queries. This verifies that the relational information about in is vital for AI-NLR, and the pair-wise degraded fuzzy set generation process introduced in Section 3.3.4 is effective to tackle with Instantiation. More experimental results about ablation study are reported in Table 6 in supplementary materials.

4.5. Case Study of Concept Representation

Entity Info
Province_of_L’Aquila A province of Italy 1.0000
Lietuvos_krepšinio_lyga A sport league in Lithuania 0.9988
Moghreb_Tétouan A sport team in Morocco 0.9988
Quebec_Route_132 A highway in Canada 0.9986
School_of_Visual_Arts A college in New York 0.9968
League_of_Ireland A sport league in Ireland 0.9777
Pančevo A city in Serbia 0.9762
Kolar A city in India 0.9755
Kemco A company in Japan 0.9740
Deyr_County A county in Iran 0.9740
Table 5. Top 10 entities with highest degrees of membership in concept “place ¡http://dbpedia.org/ontology/Place¿. The real-world places are in bold-face.

To gain insights of the fuzzy set representation of concept learned in ABIN, we present a case study on the concept “place”, ¡http://dbpedia.org/ontology/Place¿. Table 5 shows the top-10 entities ranked by their degrees of membership to . The entities of real-world places are highlighted in boldface. As 6 out of the top 10 entities are correct (real places), we believe fuzzy sets are capable of representing concepts in domain as vague sets of entities. Regarding to the other 4 entities, we observe that they are representative sport teams, leagues, or companies of a corresponding region. Although they are not real places, it makes sense that they have high degrees of membership to , as they are strongly associated to the corresponding places. Therefore, the results demonstrate that the fuzzy set based ABIN is capable of take the advantage of vagueness to explore the highly related entities.

5. Conclusion

In conclusion, we formulated the AI-MLR problem that jointly performs abductive and inductive neural logical reasoning. This is a novel problem and of great importance for users, downstream tasks, and ontological applications. The key challenges for addressing AI-NLR are the incorporation of concepts, representation of concepts, and operator on concepts. Accordingly, we propose ABIN that properly incorporates ontological axioms, represents concepts and queries as fuzzy sets, and operates on concepts based on fuzzy sets. Extensive experimental results demonstrate the effectiveness of ABIN for AI-NLR. The processed datasets and code are ready to be published to foster further research of AI-MLR.

References

  • E. Arakelyan, D. Daza, P. Minervini, and M. Cochez (2020) Complex query answering with neural link predictors. In ICLR, Cited by: §2.2, §3.2.2.
  • S. Auer, C. Bizer, G. Kobilarov, J. Lehmann, R. Cyganiak, and Z. Ives (2007) Dbpedia: a nucleus for a web of open data. In The semantic web, pp. 722–735. Cited by: §1.
  • F. Baader, I. Horrocks, C. Lutz, and U. Sattler (2017) Introduction to description logic. Cambridge University Press. Cited by: §3.2.2.
  • F. Baader, I. Horrocks, and U. Sattler (2008) Description logics. Foundations of Artificial Intelligence 3, pp. 135–179. Cited by: §3.2.1.
  • F. Baader (2003) Appendix: description logic terminology. The Description logic handbook: Theory, implementation, and applications, pp. 485–495. Cited by: §3.1.2, §3.2.1.
  • A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, and O. Yakhnenko (2013) Translating embeddings for modeling multi-relational data. NeurIPS 26. Cited by: §1.
  • J. Chen, P. Hu, E. Jimenez-Ruiz, O. M. Holter, D. Antonyrajah, and I. Horrocks (2021a) OWL2Vec*: embedding of owl ontologies. Machine Learning, pp. 1–33. Cited by: §2.3.
  • X. Chen, Z. Hu, and Y. Sun (2021b) Fuzzy logic based logical query answering on knowledge graph. arXiv preprint arXiv:2108.02390. Cited by: §2.2.
  • N. Choudhary, N. Rao, S. Katariya, K. Subbian, and C. K. Reddy (2021) Self-supervised hyperboloid representations from logical queries over knowledge graphs. In WWW, pp. 1373–1384. Cited by: §2.1.
  • R. T. Cook (2009) Dictionary of philosophical logic. Edinburgh University Press. Cited by: §1.
  • C. Elsenbroich, O. Kutz, U. Sattler, et al. (2006) A case for abductive reasoning over ontologies.. In OWLED, Vol. 216. Cited by: §1.
  • D. M. Endres and J. E. Schindelin (2003)

    A new metric for probability distributions

    .
    IEEE Transactions on Information theory 49 (7), pp. 1858–1860. Cited by: §3.3.1.
  • X. Glorot and Y. Bengio (2010) Understanding the difficulty of training deep feedforward neural networks. In AISTATS, pp. 249–256. Cited by: §3.2.1, §3.2.2.
  • T. R. Gruber (1993) A translation approach to portable ontology specifications. Knowledge acquisition 5 (2), pp. 199–220. Cited by: §1.
  • W. L. Hamilton, P. Bajaj, M. Zitnik, D. Jurafsky, and J. Leskovec (2018) Embedding logical queries on knowledge graphs. In NeurIPS, pp. 2030–2041. Cited by: §A.1, §1, §1, §2.1, §3.3.2, Table 2, §4.1.1, §4.1.1, Table 3.
  • J. Hao, M. Chen, W. Yu, Y. Sun, and W. Wang (2019) Universal representation learning of knowledge bases by jointly embedding instances and ontological concepts. In KDD, pp. 1709–1719. Cited by: §2.3.
  • J. Hao, C. J. Ju, M. Chen, Y. Sun, C. Zaniolo, and W. Wang (2020) Bio-joie: joint representation learning of biological knowledge bases. In Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, pp. 1–10. Cited by: §2.3.
  • D. P. Kingma and J. Ba (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §4.1.3.
  • E. P. Klement, R. Mesiar, and E. Pap (2004) Triangular norms. position paper i: basic analytical and algebraic properties. Fuzzy sets and systems 143 (1), pp. 5–26. Cited by: §3.2.2.
  • G. Klir and B. Yuan (1995) Fuzzy sets and fuzzy logic. Vol. 4, Prentice hall New Jersey. Cited by: §1, §2.2, §3.2.1, §3.2.2.
  • M. Kulmanov, W. Liu-Wei, Y. Yan, and R. Hoehndorf (2019) EL embeddings: geometric construction of models for the description logic el++. Cited by: §2.3.
  • M. Kulmanov, F. Z. Smaili, X. Gao, and R. Hoehndorf (2021) Semantic similarity and machine learning with ontologies. Briefings in bioinformatics 22 (4), pp. bbaa199. Cited by: §2.3.
  • R. C. Lee (1972) Fuzzy logic and the resolution principle. Journal of the ACM (JACM) 19 (1), pp. 109–119. Cited by: §3.2.1.
  • S. Liu, H. Chen, Z. Ren, Y. Feng, Q. Liu, and D. Yin (2018) Knowledge diffusion for neural dialogue generation. In ACL, pp. 1489–1498. Cited by: §1.
  • S. Mondala, S. Bhatiab, and R. Mutharajua (2021) EmEL++: embeddings for description logic. Cited by: §2.3.
  • H. Ren, W. Hu, and J. Leskovec (2019)

    Query2box: reasoning over knowledge graphs in vector space using box embeddings

    .
    In ICLR, Cited by: §1, §1, §2.1, Table 2, §4.1.1, §4.1.1, §4.1.1, §4.1.3, Table 3.
  • H. Ren and J. Leskovec (2020) Beta embeddings for multi-hop logical reasoning in knowledge graphs. NeurIPS 33. Cited by: §1, §1, §2.1, Table 2, §4.1.1, §4.1.1, §4.1.2, Table 3.
  • B. Rihoux and G. De Meur (2009) Crisp-set qualitative comparative analysis (csqca). Configurational comparative methods: Qualitative comparative analysis (QCA) and related techniques 51, pp. 33–68. Cited by: §3.2.2.
  • F. Z. Smaili, X. Gao, and R. Hoehndorf (2019) Opa2vec: combining formal and informal content of biomedical ontologies to improve similarity-based prediction. Bioinformatics 35 (12), pp. 2133–2140. Cited by: §2.3.
  • T. P. Tanon, G. Weikum, and F. Suchanek (2020) Yago 4: a reason-able knowledge base. In European Semantic Web Conference, pp. 583–596. Cited by: §1.
  • B. D. Trisedya, J. Qi, and R. Zhang (2019) Entity alignment between knowledge graphs using attribute embeddings. In AAAI, Vol. 33, pp. 297–304. Cited by: §1.
  • E. van Krieken, E. Acar, and F. van Harmelen (2020) Analyzing differentiable fuzzy implications. In KR, pp. 893–903. Cited by: §3.2.2.
  • J. Vickers (2009) The problem of induction. Stanford encyclopedia of philosophy. Cited by: §3.1.1.
  • K. Zhou, W. X. Zhao, S. Bian, Y. Zhou, J. Wen, and J. Yu (2020) Improving conversational recommender systems via knowledge graph based semantic fusion. In KDD, pp. 1006–1014. Cited by: §1.

Appendix A Supplementary Materials

a.1. Inductive Reasoning in ABIN

Here, we elaborate the method to obtain the query embedding for inductive reasoning, i.e., with parameters . We use the integrated implementation555https://github.com/snap-stanford/KGReasoning of GQE (Hamilton et al., 2018) to obtain . Specifically, the projection operation that project an entity or query embedding with relation is resolved by:

(18)

where is another query embedding that is obtained in advance or an entity embedding obtained by looking up by rows. The intersection of two query embeddings and is resolved by:

(19)

where denotes matrix concatenation over the last dimension, denote s the parameters of , and is a two-layer feed-forward network with activation. and represent the first and second attention weights, respectively. The union of two query embeddings and is resolved by:

(20)

where denotes the max operation over the last dimension.

Abduction 1p 2p 3p 2i 3i pi ip 2u up avg
w/o CC 61.4 73.8 71.8 22.9 20.1 28.8 21.6 67.5 62.6 47.8
w/o EC 58.9 69.7 68.0 42.8 39.6 41.2 22.0 66.4 62.9 52.4
ABIN 62.4 83.7 83.6 50.9 43.7 45.0 29.9 67.2 67.3 59.3
Induction 1p 2p 3p 2i 3i pi ip 2u up avg
w/o CC 20.5 22.8 23.0 18.8 20.6 14.7 25.6 10.4 18.4 19.4
w/o EC 19.4 19.1 20.5 19.1 20.1 15.0 25.7 9.1 20.3 18.7
ABIN 34.6 28.0 29.0 44.6 54.8 21.4 40.6 17.8 23.0 32.6
Table 6. Ablation Study on Subsumptions and Instantiation upon DBpedia dataset. The best Hit@3 results are in boldface.

  Algorithm 1 The learning procedure of ABIN.

 

0:  An ontological knowledge base . denotes the set of entities; denotes the set of concept (names) ; denotes the set of relations ;
0:   denotes the entity embedding matrix; denotes the concept embedding matrix; denotes the relation embedding matrix; denotes the parameters of ; denotes the parameters of .
1:  // Start training.
2:  Initialize , , , and .
3:  for each training episode do
4:     // ABDUCTION.
5:     for each query  do
6:        Sample a concept-level answer as a positive instance and non-answer concepts as negative instances;
7:        Represent as by Eq.(6);
8:        Represent as by Eq.(8), (9), (10), and (11);
9:        Calculate by Eq.(12) and (13) given and ;
10:        for each negative instance  do
11:           Represent as by Eq.(6);
12:           Calculate by Eq.(12) and (13) given and ;
13:        end for
14:     end for
15:     // INDUCTION.
16:     for each query  do
17:        Sample an entity-level answer as a positive instance and non-answer entities as negative instances;
18:         Look up by rows;
19:         ;
20:        Calculate by Eq.(14) given and ;
21:        for each negative instance  do
22:            Look up by rows;
23:           Calculate by Eq.(14) given and ;
24:        end for
25:     end for
26:     // SUBSUMPTION.
27:     for each pair of concepts  do
28:        Sample concepts as negative instances;
29:        , Look up by rows;
30:        Calculate by Eq.(15) given and ;
31:        for each negative instance  do
32:            Look up by rows;
33:           Calculate by Eq.(15) given or with equal probability;
34:        end for
35:     end for
36:     // INSTANTIATION.
37:     for each pair of concept and entity  do
38:        Sample negative concepts ;
39:        Sample negative entities ;
40:         Look up by rows;
41:         Look up by rows;
42:        Calculate by Eq.(16) given and ;
43:        for each negative concept  do
44:            Look up by rows;
45:           Calculate by Eq.(16) given ;
46:        end for
47:        for each negative entity  do
48:            Look up by rows;
49:           Calculate by Eq.(16) given ;
50:        end for
51:     end for
52:     Calculate by Eq.(17);
53:     Update ; Update ; Update ; Update ; Update ;
54:  end for
55:  return updated , , , and .