1. Introduction
Along with the rapid development of highquality largescale knowledge infrastructures (Tanon et al., 2020; Auer et al., 2007)
, researchers are increasingly interested in exploiting knowledge bases for realworld applications, such as knowledge graph completion
(Bordes et al., 2013) and entity alignment (Trisedya et al., 2019). However, to take advantage of knowledge bases, a fundamental yet challenging task still remains unsolved, i.e., neural logical reasoning (NLR), which attempts to answer complex structured queries that include logical operations and multihop projections given the facts in knowledge bases with distributed representations (Hamilton et al., 2018). Recently, efforts (Hamilton et al., 2018; Ren et al., 2019; Ren and Leskovec, 2020) have been made to develop NLR systems by designing strategies to learn geometric or uncertaintyaware distributed query representations, and proposing mechanisms to deal with various logical operations on these distributed representations.However, existing neural logical reasoners cannot fully fulfill our needs. In many realworld scenarios, we not only expect specific entitylevel answers, but also seek for more descriptive conceptlevel answers, where each of the concepts is a summary of a set of entities. For example, as shown in Figure 1, the query asks “who will be interested in something that G. Hinton is investigating and Google is deploying?”, users expect entitylevel answers like Meta, Amazon, MIT, and Y. LeCun, as well as conceptlevel answers such as AI Researchers, The Academia, and The Industry. In this example, the conceptual answer The Academia refers to a summary of a set consisting of Y. LeCun and MIT, and it is intuitively desirable for users and worth exploring. In biomedical applications, people may want to find the causes for a set of symptoms and expect both entitylevel answers (such as SARSCoV2 causing Fever) as well as conceptlevel answers (such as Viral infections causing Fever). In this case, the answer constitutes a descriptive conceptlevel answer (e.g., Viral infections) that is a summary of a set of entitylevel answers. Downstream tasks like online chatbots (Liu et al., 2018) and conversational recommender systems (Zhou et al., 2020) also need to retrieve rich and comprehensive answers to provide better services. Thus, providing both entitylevel and conceptlevel answers can highly improve their capability of generating more informative responses to users and enriching the semantic information in answers for downstream tasks.
From the perspective of logic theory, the ability of providing both conceptlevel and entitylevel answers corresponds to the capability of jointly supporting abductive and inductive neural logical reasoning (AINLR). That is to say, previous NLR systems support inductive reasoning, while more general AINLR systems additionally support abductive reasoning which is highly useful in the context of query answering over ontologies (Elsenbroich et al., 2006). Specifically, neural inductive reasoners seek to infer extensional answers by listing the elements of the set of answers, while neural abductive reasoners aim at inferring explanations from given observations in the form of concepts, and therefore are providing intensional answers. In other words, neural inductive reasoning infers the set of entities that satisfies each query, while neural abductive reasoning infers explanations of each query by summarizing the entitylevel answers with descriptive concepts. Therefore, the AINLR problem is more general than the regular NLR problem in terms of including not only induction that gives extensional entitylevel answers, but also abduction that yields intensional conceptlevel answers as explanations. Note that such descriptive concepts are higherlevel abstractions of the set of entities and are more informative than the set in some cases (Cook, 2009).
From the perspective of users and downstream tasks, AINLR is helpful by jointly providing the more informative conceptlevel and entitylevel answers. From the perspective of logic theory, AINLR is useful in ontological applications by jointly supporting abductive and inductive reasoning. However, existing methods (Hamilton et al., 2018; Ren et al., 2019; Ren and Leskovec, 2020) can hardly reach AINLR for the following reasons. On the one hand, concepts are excluded from the NLR systems. That is to say, previous NLR systems perform reasoning upon regular knowledge graphs where only entities and relations exist. On the other hand, mechanisms for exploiting concepts have not been well established. Specifically, previous solutions only measure queryentity similarity for NLR, without considering concept representations and operators involving concepts, such as query–concept similarity for neural abductive reasoning.
Along this line, we propose an original solution named ABIN for AINLR. ABIN is a shortname which stands for an ABduction and INduction reasoner. The key challenges for addressing AINLR are the incorporation of concepts, representation of concepts, and operator on concepts. First, we observe that ontologies include taxonomic hierarchies of concepts, concept definitions, and concept subsumption relations (Gruber, 1993). To incorporate concepts into the AINLR system, we thus introduce some description logic based ontological axioms into the system to provide sources of concepts. Second, we find that fuzzy sets (Klir and Yuan, 1995), i.e., sets whose elements have degrees of membership, can naturally bridge entities with concepts, i.e., vague sets of entities. Therefore, we represent concepts as fuzzy sets in ABIN. Meanwhile, properly representing queries is the prerequisite of effectively operating on concepts. We find that fuzzy sets can also bridge entities with queries, i.e., vague sets of entitylevel answers. The theoreticallysupported, vague, and unparameterized fuzzy set operations enable us to resolve logical operations within queries. Thus, the adoption of fuzzy sets is an ideal solution for concepts and queries representation in AINLR. Then operators involving concepts can also be designed based on fuzzy sets, including queryconcept operators for abduction, entityconcept operators for instantiation, and conceptconcept operator for subsumption. Attributed to the well designed operators, a joint abductive and inductive neural logical reasoner can be achieved.
We summarize the main contribution of this work as follows:

To the best of our knowledge, we are the first to focus on the AINLR problem that aims at providing both entitylevel and conceptlevel answers so as to jointly achieve inductive and abductive neural logical reasoning, which better satisfies the need of users, downstream tasks, and ontological applications;

We propose an original solution ABIN that properly incorporates, represents, and operates on concepts. We incorporate ontologies to provide sources of concepts and employ fuzzy sets as the representations of concepts and queries. Logical operations are supported by the wellestablished fuzzy set theory and operators involving concepts are rationally designed upon fuzzy sets;

We conduct extensive experiments and demonstrate the effectiveness of ABIN for AINLR. We publish in public two preprocessed benchmark datasets for AINLR and the implementation of ABIN to foster further research^{1}^{1}1https://github.com/lilv98/ABIN.
2. Related Work
2.1. Neural Logical Reasoning
Given the vital role of neural logical reasoning (NLR) in knowledge discovery and artificial intelligence, great efforts have been made to develop sophisticated NLR systems in recent years. GQE (Hamilton et al., 2018) is the pioneering work in this field, the authors formulate the NLR problem and propose to simply use points in the embeddings space to represent logical queries. Q2B (Ren et al., 2019) claimed that the representation of each query in the embedding space should be a geometric region instead of a single point because each query is equivalent to a set of entitylevel answers in the embedding space. Therefore, they use hyperrectangles that can include multiple points in the embedding space to represent queries. HypE (Choudhary et al., 2021) extended Q2B by using hyperboloids in a Poincaré ball for distributed query representation. However, these methods can only deal with projection, intersection, and union, without considering negation. More recently, BetaE (Ren and Leskovec, 2020)
proposed to use Beta distributions over the embedding space to represent queries for NLR. The advantage of using distributions over points or geometric regions is that they can properly handle queries with negations and queries without answers. However, these reasoners could only give extensional entitylevel answers, while we focus on the more general AINLR problem that aims at additionally providing descriptive concepts to summarize the entitylevel answers. We bring neural logical reasoners to the stage of abductive and inductive reasoning.
2.2. Fuzzy Logic for NLR
Besides representing logical queries as points, geometric reagions, or distributions, more recent methods explore fuzzy logic (Klir and Yuan, 1995) for NLR. CQD (Arakelyan et al., 2020) used norm and conorms from the fuzzy logic theory to achieve high performance on zeroshot settings. More specifically, mechanisms are proposed for the inference stage on various types of queries, while only training the simple neural link predictor on triples (1p queries in Figure 2). FuzzQE (Chen et al., 2021b)
directly represented entities and queries using embeddings with specially designed restrictions and interpreted them as fuzzy sets for NLR. However, this study still focuses on the regular NLR problem, while we are solving a more general problem that additionally includes neural abductive reasoning. Furthermore, we explicitly include concepts and represent them as fuzzy sets, whereas FuzzQE represents only the query as fuzzy set. Moreover, CQD only uses fuzzy logic at the entity level and FuzzQE uses fuzzy sets with arbitrary numbers of elements as the tunable embedding dimension without reasonable interpretations. We interpret queries as fuzzy sets where each element represents the probability of an entity being an answer, aligning with the definition and the essence of fuzzy sets
(Klir and Yuan, 1995), i.e., sets where each element has a degree of membership. This allows us to fully exploit fuzzy logic and provides a theoretical foundation in fuzzy set theory for our work.2.3. Ontology Representation Learning
Recently, several methods that exploit ontologies from the perspective of distributed representation learning have been developed (Kulmanov et al., 2021). ELEm (Kulmanov et al., 2019) and EmEL (Mondala et al., 2021) learn geometric embeddings for concepts in ontologies. The key idea of learning geometric embeddings is that the embedding function projects the symbols used to formalize axioms into an interpretation of these symbols such that is a model of the ontology. Other approaches (Smaili et al., 2019; Chen et al., 2021a) rely on regular graph embeddings or word embeddings and apply them to ontology axioms. Another line of research (Hao et al., 2019, 2020) focuses on jointly embedding entities and relations in regular knowledge graphs, as well as concepts and roles (relations) in ontological axioms. Our work is related to ontology representation learning in that we incorporate some description logic based ontological axioms in Section 3.1.2 to provide sources of concepts, and we exploit concepts with distributed representation learning in our proposed ABIN for AINLR. Methods for representation learning with ontologies have previously only been used to answer link prediction tasks such as predicting protein–protein interactions or performing knowledge graph completion, which can be viewed as answering 1p queries in Figure 2 whereas we also focus on more complex queries as well as providing abductive conceptlevel answers.
3. Methodology
Incorporating, representing, and operating on concepts are the three key components for an abductive and inductive neural logical reasoner. In this section, we first formulate the AINLR problem along with the process of incorporating concepts into the reasoning system. Then we propose an original solution ABIN for AINLR by designing concept representations and operators involving concepts. We introduce optimization and inference procedures in the end.
3.1. Incorporating Concepts
3.1.1. Regular NLR
The regular NLR problem is defined on knowledge graphs. A knowledge graph is formulated as , where , , denote the head entity, relation, and tail entity in triple , respectively, and refer to the entity set and the relation set in .
In the context of NLR, as shown in Figure 2.(a), each triple is regarded as a positive sample of the 1p query with an answer that satisfies , where is the anchor entity and is the projection operation with relation . Furthermore, the regular NLR problem may also address the intersection, union, and negation operations , , and within queries. Thus, infinite types of queries can be found with the combinations of these logical operations. We consider the representative types of queries, which are listed and demonstrated with their graphical structures in Figure 2. For example, queries of type pi in Figure 2.(c) are to ask .
Regular neural logical reasoners seek to provide extensional entitylevel answers for each query. In particular, the answers are a set of entities that satisfies the query by inductive reasoning. Following the definition of inductive reasoning (Vickers, 2009), the given knowledge graph might give us very good reason to accept that each element of the set is an answer of the query, but it does not ensure that. Therefore, we predict the possibility of each candidate entity satisfying a query . We then rank the possibilities and select the top entities in as the set of answers. Since all the candidate answers are entities, we can only retrieve extensional entitylevel answers from the regular NLR systems.
3.1.2. Abductive and Inductive NLR
A joint abductive and inductive neural logical reasoner is upon ontological axioms provided in a knowledge base
, which is an ordered pair (
, ) for TBox and ABox , where is a finite set of terminological axioms and is a finite set of assertion axioms. Specifically, terminological axioms within a TBox are of the form where the symbol denotes subsumption (). In general, and can be concept descriptions that consist of concept names, quantifiers and roles (relations), and logical operators; we limit ABIN to axioms where and are concept names that will not involve roles or logical operators (Baader, 2003). In the followings, we do not distinguish between a concept name and a concept description unless there are special needs. Then, a TBox is:(1) 
where denotes the set of concept names in . accounts for the source of concepts and the pairwise concept subsumption information in the AINLR system. Assertion axioms in consist of two parts. The one part is the role assertion that is expressed as:
(2) 
where denote entities, denotes the entity set in , denotes the role assertion between and , and is the the role set of . accounts for the triplewise relational information about entities and roles in AINLR. The other part within is the concept instantiation between an entity and an concept :
(3) 
where represents is an instance of . serves as the bridge between and by providing pairwise links between entities and concepts.
Since we incorporate concepts in the AINLR systems, we are able to ask questions about concepts. In particular, for a query of arbitrary type discussed in Section 3.1.1, we not only perform inductive reasoning that returns a set of entities of as the extensional entitylevel answers, but also we perform abductive reasoning that infers an explanation for each query result by summarizing entitylevel answers with descriptive concepts, yielding another set of conceptlevel answers as the intensional conceptlevel answers. More specifically, as shown in Figure 2, the answers are no longer restricted to be (represented as circles), they can also be (represented as squares). To achieve this goal, we predict the possibility of each candidate entity as well as the possibility of each candidate concept satisfying a query . We then rank predicted scores of candidate entities and predicted scores of candidate concepts. We select and combine the top results from each set of candidates as the final answers of with extensional entitylevel and intensional conceptlevel answers .
Note that the regular NLR problem is a subproblem of the AINLR problem. On the one hand, regular NLR systems can only provide a subset of the answers provided by abductive and inductive neural logical reasoners, i.e., . On the other hand, the entire in the context of regular NLR is equivalent to in the case of the AINLR problem, which is a subset of the ontological knowledge base, i.e., , leaving and with conceptual information in the ontologies not explored. Therefore, the problem we investigate is more general in terms of providing more answers and reasoning over more complex knowledge bases.
3.2. Representing Concepts and Queries
In this subsection, we first introduce how to represent concepts as fuzzy sets in our proposed ABIN for AINLR. Then we represent queries as fuzzy sets as well to prepare for the later operations that involve concepts and queries.
3.2.1. Representing Concepts
We are motivated to represent concepts as fuzzy sets by the relationship between concepts and entities. We gain insights on such relationship from the basic definition of semantics in description logics (Baader et al., 2008):
Definition 1 ().
A terminological interpretation over a signature consists of:

a nonempty set called the domain

an interpretation function that maps:

every entity to an element

every concept to a subset of

every role (relation) to a subset of

As we use a functionfree language (Baader, 2003), we set to be the Herbrand universe (Lee, 1972) of our knowledge base, i.e., . Therefore, according to Definition 1, the interpretation of concept is a subset of , which is finite. On the other hand, fuzzy sets (Klir and Yuan, 1995) over the Herbrand Universe are finite sets whose elements have degrees of membership:
(4) 
where is the membership function that measures the degree of membership of each element. Therefore, we further interpret all concepts as fuzzy sets over the finite domain as the elements of fuzzy sets . Thus, we have:
(5) 
As the Herbrand universe for our language is always finite, the interpretation of concept is fully determined by the fuzzy membership function that assigns a degree of membership to each entity for , where and are the interpretation of the entity (name) set and the concept (name) set.
To obtain the degree of membership of entity in , i.e., , we first randomly initialize the embedding matrix of concepts and entities as and with Xavier uniform initialization (Glorot and Bengio, 2010), where is the embedding dimension. Then we obtain the embedding of each concept by looking up the rows of . The embedding then serves as the generator of the fuzzy set representation of each concept . Thus, we compute the similarities between each concept and every entity in our universe as the degrees of membership of each entity in the fuzzy set:
(6) 
where symbol denotes matrix multiplication and represents the matrix transposition. The measured similarities are then normalized to
using the bitwise sigmoid function
. Here, the setwise operation to obtain consists of pairwise operations on the entity–concept pairs; we use the same operator for Instantiation, which we will explain in Section 3.3.4.3.2.2. Representing Queries
Properly representing queries is the prerequisite of operating on concepts. Fuzzy sets are particularly suitable to represent not only concepts, but also queries, because interpretations of queries are essentially interpretations of concepts. More accurately, queries correspond to concept descriptions that may include concept names, roles (relations), quantifiers, and logical operations. We can use the same formalism designed for representing concepts to represent entities, i.e., as a special type of fuzzy sets (Rihoux and De Meur, 2009) that assigns the membership function to for exactly one entity and to to all others. Consequently, we can interpret entities as concepts. As explained in section 3.1.1, queries may consist of entities, relations, and logical operations. Therefore, queries are interpreted as concept descriptions and we regard entities within queries as singleton concepts. Thus, we can use the same description logic semantics (Baader et al., 2017) to interpret a query and concept in Definition 1: an interpretation function maps every query to a subset of . As the Herbrand universe is finite, the interpretation of query is fully determined by the fuzzy membership function
(7) 
Besides, representing queries as fuzzy sets have other advantages. Firstly, fuzzy logic theory (Klir and Yuan, 1995) wellequipped us to interpret logical operations within queries as the vague and unparameterized fuzzy set operations. The preservation of vagueness is important in that we are performing inductive and abductive reasoning that requires uncertainty, rather than deductive reasoning that guarantees the correctness. Unparameterized operations are desirable because they require fewer data during training and are often more interpretable. Secondly, since concepts are already represented as fuzzy sets, it would be more convenient for us to employ the same form of representation and retain only one form of representation within the AINLR system. We explain how to represent queries as fuzzy sets in detail as the followings.
Representing Atomic Queries
Each multihop logical query consists of one or more Atomic Queries (AQ), where an AQ is defined as a query that only contains projection(s) from an anchor entity without logical operations such as intersection , union , and negation . Therefore, the first step to represent queries is to represent AQs. We obtain the embeddings of each entity and the relation by looking up the rows of the randomly initialized entity embedding matrices and with Xavier uniform initialization (Glorot and Bengio, 2010). Then, the generator for fuzzy set representation of an valid AQ is . Thus, we obtain the fuzzy set corresponding to the query as:
(8) 
Similar to the process of obtaining fuzzy set representations of concepts, Eq.(8) is to aquire the degrees of membership of every candidate being an answer to a given AQ by computing their normalized similarities.
Fusing Atomic Queries
AQs are fused by logical operations to form multihop logical queries. Since AQs are already represented in fuzzy sets and we are equipped with the theoretically supported fuzzy set operations, we interpret logical operations as fuzzy set operations over concepts to fuse AQs into the final query representations.
For two fuzzy sets in domain : and , we have the intersection over the two fuzzy sets as:
(9) 
the union over the two fuzzy sets as:
(10) 
and we have the negation over as:
(11) 
where a norm is a generalisation of conjunction in logic (Klement et al., 2004). Some examples of norms include the Gödel norm , the product norm , and the Łukasiewicz norm (van Krieken et al., 2020). Analogously, a conorm is dual to norm and generalizes logical disjunction – given a norm , the complementary conorm is defined by (Arakelyan et al., 2020). The choice of the
norm is a hyperparameter of ABIN.
Thus, each query can be decomposed into AQs and represented as a fuzzy set with Eq.(8), and then fuzzy set representations of AQs are fused by the fuzzy set operations in Eq.(9), (10), and (11) to obtain the final representation of the query. Note that fuzzy set operations hold the property of closure, which means the input and output of these operations remain fuzzy sets. Thus, the final representation of each query is also a fuzzy set .
3.3. Operating on Concepts
In previous sections, we manage to prepare for designing operators involving concepts by representing concepts and queries in fuzzy sets. Here, we design operators involving concepts for abduction, induction, subsumption, and instantiation.
3.3.1. Abduction
Abductive reasoning is to give explanations for a set of observations; here, the answers to a query are the observations that we summarize by descriptive concepts, i.e., provide as discussed in Section 3.1.2. We measure the possibility of each being an intensional conceptlevel answer of a given query upon fuzzy set representations. More specifically, we measure the similarity between and based on the JensenShannon divergence (Endres and Schindelin, 2003)
, which is a symmetrized and smoothed version of the KullbackLeibler divergence
. The similarity function we use for abductive inference is defined by:(12) 
where , and represent the normalized fuzzy set representations of the considered query and concept descriptions, which are given by:
(13) 
where is a small value to avoid division by zero and is the exponent value in the norm formulation . are then used for model training and abductive inference in Section 3.4.
3.3.2. Induction
Inductive reasoning aims to provide extensional entitylevel answers; for this purpose, only query–entity similarities need to be measured without the necessity of designing new mechanisms. Therefore, we follow the pioneering work (Hamilton et al., 2018) on NLR and to represent each query as an embedding and measure query–entity similarity for inductive reasoning:
(14) 
where is the margin, denotes the function to obtain query embedding , and denotes the parameters of . We explain in detail in Section A.1.
Partition  Abd1p  Abdother  Ind1p  Indother  Sub  Ins  NLR1p  NLRother  
YAGO4  32,465  8,382  75  #Train  189,338  10,000  101,417  10,000  16,644  83,291  184,708  10,000 
#Valid/#Test  1,000/1,000  1,000/1,000  1,000/1,000  1,000/1,000      1,000/1,000  1,000/1,000  
Dbpedia  28,824  981  327  #Train  473,924  10,000  136,821  10,000  2,582  225,436  362,257  10,000 
#Valid/#Test  1,000/1,000  1,000/1,000  1,000/1,000  1,000/1,000      1,000/1,000  1,000/1,000  
3.3.3. Subsumption
As defined by Eq.(1), supplies for relational information among concepts with the form of concept subsumptions. Although concepts are represented in fuzzy sets and we already designed mechanism to measure the similarity between two fuzzy sets, we can not directly apply the method in Section 3.3.1
for concept subsumptions. It is because we need to measure the degree of inclusion of one concept to another instead of the similarities between them. The degree of inclusion is asymmetrical and more complex than the similarity measurement. Therefore, we employ a neural network
to model the degree of inclusion:(15) 
where symbol denotes matrix concatenation over the last dimension, and denotes the parameters of . In this paper, is a twolayer feedforward network with activation. Note that we directly use the embeddings of concepts without interpreting concept in the Herbrand universe of entities because neither conceptentity relationships need to be modeled nor logical operations need to be resolved.
3.3.4. Instantiation
As defined by Eq.(3), bridges and by providing links between entities and concepts. Such links instantiate concept with its describing entities and thus offer relational information with the form of concept instantiation. Recall that in Section 3.2.1, we obtain the fuzzy set representation of concepts by computing the similarities between the given and every candidate with Eq.(6). In the case of concept instatiation, the setwise computation Eq.(6) is degraded to pairwise similarity measurement for each conceptentity pair:
(16) 
where and are the categorical embeddings of concept and entity , respectively.
3.4. Optimization
The parameters to optimize in our model ABIN include the entity embedding matrix , the concept embedding matrix for the basic representation of concepts that is out of domain , the relation embedding matrix , in Section 3.3.3, and in Section A.1. In the training stage, we sample negative samples for each positive instance of abductive reasoning by corrupting with randomly sampled (). Similarly, negative samples for induction are obtained by corrupting in with randomly sampled . For subsumption and instantiation, both sides of the conceptconcept pairs and conceptentity pairs are randomly corrupted following the same procedure.
The loss of ABIN is defined as
(17) 
where denotes the set of the four included task discussed Section 3.3, (or ) denotes the predicted similarity or degree of inclusion of the positive (or negative) sample according to task . The overall optimization process of is outlined in Algorithm 1 in supplementary materials.
In the inference stage, we predict (or ) for every candidate concept (or entity ) regarding to query and select the top results to be the intensional conceptlevel answers (or extensional entitylevel answers ) for query . Thus, we are able to achieve AINLR by providing the comprehensive answers . Although subsumption in Section 3.3.3 and instantiation in Section 3.3.4 are not included in the inference stage, they empowered ABIN to better represent and operate concepts by providing training instances and extra supervision signals.
MRR  Hit@3  
322  1p  2p  3p  2i  3i  pi  ip  2u  up  avg  1p  2p  3p  2i  3i  pi  ip  2u  up  avg  
YAGO4  GQE (Hamilton et al., 2018)  35.3  49.5  33.7  51.3  43.9  11.8  9.1  14.4  6.6  28.4  43.7  67.4  44.9  67.1  49.9  15.0  12.3  19.7  9.4  36.6 
Q2B (Ren et al., 2019)  37.3  53.4  59.6  55.0  47.6  2.1  1.6  1.7  1.1  28.8  47.1  75.4  78.2  70.0  58.2  2.8  1.9  1.9  1.4  37.4  
BetaE (Ren and Leskovec, 2020)  39.0  57.0  58.9  52.9  45.8  10.4  8.9  2.7  6.5  31.3  47.8  74.7  76.7  64.2  52.5  10.1  10.2  2.9  5.0  38.2  
ABIN  51.3  76.4  82.7  55.9  53.9  51.3  48.9  54.3  45.9  57.8  60.3  88.8  88.7  66.4  65.5  60.2  57.1  59.9  52.5  66.6  
DBpedia  GQE (Hamilton et al., 2018)  27.1  35.5  32.5  30.5  32.0  14.0  14.7  9.8  11.8  23.1  28.7  42.9  40.0  32.5  36.0  13.0  14.4  7.2  10.7  25.0 
Q2B (Ren et al., 2019)  26.4  35.7  32.6  30.4  29.9  13.5  14.4  10.3  11.2  22.7  28.2  41.4  38.0  32.7  33.5  11.7  13.0  8.4  9.3  24.0  
BetaE (Ren and Leskovec, 2020)  30.4  38.7  40.0  32.9  34.2  14.8  11.4  5.6  9.0  24.1  34.1  45.4  50.8  37.2  41.2  14.6  10.9  4.0  8.0  27.4  
ABIN  55.0  80.8  80.7  42.9  36.7  42.0  28.5  63.4  64.9  55.0  62.4  83.7  83.6  50.9  43.7  45.0  29.9  67.2  67.3  59.3  
4. Experiments
In this section, we conduct extensive experiments to answer the
following research questions:
RQ1 How to properly compare ABIN with methods that do not support abduction?
RQ2 How does
ABIN perform for abductive reasoning?
RQ3 How does ABIN
perform for inductive reasoning?
RQ4 How do the introduced
subsumption and instantiation perators affect the performance of ABIN?
4.1. Experimental Settings
4.1.1. Baselines (RQ1)
The considered baseline methods are the three most established methods in NLR, namely GQE (Hamilton et al., 2018), Q2B (Ren et al., 2019), and BetaE (Ren and Leskovec, 2020), which employ points, geometric regions, and distributions to represent queries, respectively. Since the regular neural logical reasoners can only provide extensional entitylevel answers with inductive reasoning, we need to come up with a way to make them give conceptlevel answers without explicitly trained on abduction instances, so as to be compared with our proposed ABIN on abductive reasoning.
Therefore, we introduce the Onemorehop experiment to serve as baselines for the abduction task. That is, we exploit all the information given by and simply degrade concepts to entities in the training stage. Specifically, we first augment by the transductive links provided by . Then we combine the augmented and to form the new knowledge graph . Note that part of the entities in are the degraded concepts and contains an additional relation to describe the isInstanceOf relationship between an entity and a concept. Thus, we construct training examples of various types of queries using and update model parameters following (Ren et al., 2019).
In the inference stage, two sets of candidate entities are prepared for each query. One of them is the regular entitylevel candidate set for inductive reasoning, which can be ranked following the original papers (Hamilton et al., 2018; Ren et al., 2019; Ren and Leskovec, 2020). The other set contains the degraded concepts for abductive reasoning. To predict the possibility of a concept being and intensional answer of a query , we add one more projection operation with the relation , so as to construct the abductive query: . In other words, abductive reasoning is implicitly achieved by an additional hop asking the isInstanceOf upon inductive queries, i.e., the Onemorehop.
4.1.2. Datasets
We conduct experiments on two commonlyused realworld largescale knowledge bases, namely YAGO4 and DBpedia. Specifically, we use English Wikipedia version^{2}^{2}2https://yagoknowledge.org/downloads/yago4 of YAGO4 and 201610 release^{3}^{3}3http://downloads.dbpedia.org/wikiarchive/downloads201610.html of DBpedia. To preprocess the dataset for the AINLR problem, we first filter out lowdegree entities in and with the threshold 5. Then we split to two sets with the ratio 95% and 5% for training and evaluation, respectively. We use the same procedure as BetaE (Ren and Leskovec, 2020) to construct instances of logical queries from . We use all the triples in in the training set as training examples of 1p queries and randomly select certain amount of training and evaluation examples for each of the other types of queries as stated in Table 1. We then split the evaluation set of each type of queries to the validation set and the testing set. We summarize the statistics of the preprocessed datasets in Table 1, where Abd, Ind, Sub, Ins, and Base represent the instances for abduction, induction, subsumption, instantiation, and the NLR baselines, respectively. The processed datasets along with the code for preprocessing will be published in public to foster further research.
MRR  Hit@3  
322  1p  2p  3p  2i  3i  pi  ip  2u  up  avg  1p  2p  3p  2i  3i  pi  ip  2u  up  avg  
YAGO4  GQE (Hamilton et al., 2018)  25.8  15.8  4.6  26.4  28.4  21.7  18.1  9.6  18.2  18.7  29.7  17.3  4.6  29.0  31.4  23.5  18.6  13.6  17.4  20.6 
Q2B (Ren et al., 2019)  24.5  17.2  6.8  25.7  28.8  24.5  20.2  8.6  17.7  19.3  28.4  20.0  7.0  29.5  34.4  27.0  21.3  11.6  18.5  22.0  
BetaE (Ren and Leskovec, 2020)  28.2  19.7  9.5  30.1  33.4  28.3  22.0  10.6  20.7  22.5  31.4  22.3  12.0  33.5  37.5  31.3  24.1  12.5  23.8  25.4  
ABIN  34.9  23.8  15.1  50.9  61.6  31.6  35.3  13.9  21.3  32.0  39.6  27.3  18.4  56.6  70.2  34.2  39.1  14.9  23.7  36.0  
DBpedia  GQE (Hamilton et al., 2018)  19.6  16.1  18.2  31.4  38.2  16.8  28.5  10.1  18.0  21.9  25.2  18.9  19.8  26.5  43.6  17.2  33.3  12.6  21.1  24.2 
Q2B (Ren et al., 2019)  16.3  13.7  15.4  22.6  28.5  18.1  22.8  7.4  12.9  17.5  21.3  16.5  17.7  27.5  31.9  19.7  25.7  9.1  15.4  20.7  
BetaE (Ren and Leskovec, 2020)  20.2  20.1  19.3  25.4  29.7  24.2  25.2  12.3  24.2  22.3  20.9  23.2  21.7  27.5  32.9  27.5  28.4  13.0  25.1  24.5  
ABIN  28.8  24.5  24.4  38.4  46.3  20.1  33.6  14.0  20.6  27.9  34.6  28.0  29.0  44.6  54.8  21.4  40.6  17.8  23.0  32.6  
4.1.3. Implementation Details
We implement ABIN using PyTorch
^{4}^{4}4https://pytorch.org/ and conduct all the experiments on Linux server with GPUs (Nvidia RTX 3090) and CPU (Intel Xeon). In the training stage, the initial learning rate of the Adam (Kingma and Ba, 2014) optimizer, the embedding dimension , and the batch size, are tuned by grid searching within {, , }, {128, 256, 512}, and {256, 512, 1024}, respectively. We keep the number of corrupted negative samples for each positive sample , the small value , the exponent value , the margin , and the adopted type of norm as 4, , 1, 12, and , respectively. We employ early stop with validation interval of 50 and tolerance of 3 for model training. In the test phase, following (Ren et al., 2019), we use the filtered setting and report the averaged results of Mean Reciprocal Rank (MRR) and Hits@3 over 3 independent runs.4.2. Abductive Reasoning (RQ2)
We conduct the Onemorehop experiment as described in Section 4.1.1 to answer RQ2. As shown in Table.2
, our proposed ABIN consistently outperforms baseline methods on various evaluation metric with large margins. For the
basic queries summarized in Figure 2 that are simply projections and intersections, our proposed ABIN significantly improved the performance of abductive reasoning, especially for the multiple projection queries 1p, 2p, and 3p. For extra queries in Figure.2 that are more complex in terms of including unions or combined logical operations, we even boosted the performance exponentially. The average performance of ABIN is also significantly better than baseline methods.The superior performance of ABIN can be explained for two reasons. First, due to the lack of abductive reasoning capabilities, GQE, Q2B, and BetaE need to do reasoning over more complicated queries. For example, baseline methods need to do reasoning over an ipp query to provide intensional conceptlevel answers of an ip query . Therefore, 1p queries become 2p queries for baseline methods, 2p becomes 3p, and so on. Thus, the complexity of the transformed queries limits the baseline performance. Second, explicit supervision signals for abductive reasoning are not provided by the baselines. That is to say, since the concepts are degraded as entities, regular NLR methods could not explicitly feed the empirical error on conceptlevel answers back to update the model parameters. It is thus understandable that the baseline methods cannot perform well on abductive reasoning to provide intensional conceptlevel answers, especially on extra queries that are more complicated and require supervision signals more eagerly.
4.3. Inductive Reasoning (RQ3)
We formulate the AINLR problem as to jointly provide intensional conceptlevel answers, i.e., abductive reasoning, as well as extensional entitylevel answers, i.e., inductive reasoning. Although ABIN is designed for AINLR, it is interesting to know the performance of ABIN on inductive reasoning only (for answering RQ3). The results of inductive reasoning tasks in Table 3 show that ABIN also outperforms regular NLR methods on most types of queries on various metrics. The performance gain of ABIN is credited to its capability of representing and operating on concepts. It thus has additional information of the relationships among queries, entities, and concepts, which are helpful for inductive reasoning.
Abduction  1p  2p  3p  2i  3i  pi  ip  2u  up  avg 

w/o Sub  53.3  72.4  71.5  19.0  15.8  24.1  19.3  59.9  61.7  44.1 
w/o Ins  52.8  68.8  67.7  38.3  35.4  34.2  19.3  56.2  61.6  48.3 
ABIN  55.0  80.8  80.7  42.9  36.7  42.0  28.5  63.4  64.9  55.0 
Induction  1p  2p  3p  2i  3i  pi  ip  2u  up  avg 
w/o Sub  17.7  20.1  21.0  18.0  18.8  14.9  22.7  9.1  16.0  17.6 
w/o Ins  17.4  18.8  19.0  18.1  18.7  14.3  22.9  8.7  17.7  17.3 
ABIN  28.8  24.5  24.4  38.4  46.3  20.1  33.6  14.0  20.6  27.9 
4.4. Ablation Study (RQ4)
We conduct ablation study on DBpedia dataset to answer RQ4. As shown in Table 4, when the Subsumption task is not included, i.e., ontological axioms in are not used and is not computed, ABIN w/o Sub underperforms ABIN on all types of queries for both abductive and inductive reasoning. Such results clearly demonstrate the importance of the relational information between concepts to be used for AINLR, and the effectiveness of the designed operator in Section 3.3.3 for handling such information. On the other hand, ABIN consistently outperforms w/o Ins on all types of queries. This verifies that the relational information about in is vital for AINLR, and the pairwise degraded fuzzy set generation process introduced in Section 3.3.4 is effective to tackle with Instantiation. More experimental results about ablation study are reported in Table 6 in supplementary materials.
4.5. Case Study of Concept Representation
Entity  Info  
Province_of_L’Aquila  A province of Italy  1.0000 
Lietuvos_krepšinio_lyga  A sport league in Lithuania  0.9988 
Moghreb_Tétouan  A sport team in Morocco  0.9988 
Quebec_Route_132  A highway in Canada  0.9986 
School_of_Visual_Arts  A college in New York  0.9968 
League_of_Ireland  A sport league in Ireland  0.9777 
Pančevo  A city in Serbia  0.9762 
Kolar  A city in India  0.9755 
Kemco  A company in Japan  0.9740 
Deyr_County  A county in Iran  0.9740 
To gain insights of the fuzzy set representation of concept learned in ABIN, we present a case study on the concept “place”, ¡http://dbpedia.org/ontology/Place¿. Table 5 shows the top10 entities ranked by their degrees of membership to . The entities of realworld places are highlighted in boldface. As 6 out of the top 10 entities are correct (real places), we believe fuzzy sets are capable of representing concepts in domain as vague sets of entities. Regarding to the other 4 entities, we observe that they are representative sport teams, leagues, or companies of a corresponding region. Although they are not real places, it makes sense that they have high degrees of membership to , as they are strongly associated to the corresponding places. Therefore, the results demonstrate that the fuzzy set based ABIN is capable of take the advantage of vagueness to explore the highly related entities.
5. Conclusion
In conclusion, we formulated the AIMLR problem that jointly performs abductive and inductive neural logical reasoning. This is a novel problem and of great importance for users, downstream tasks, and ontological applications. The key challenges for addressing AINLR are the incorporation of concepts, representation of concepts, and operator on concepts. Accordingly, we propose ABIN that properly incorporates ontological axioms, represents concepts and queries as fuzzy sets, and operates on concepts based on fuzzy sets. Extensive experimental results demonstrate the effectiveness of ABIN for AINLR. The processed datasets and code are ready to be published to foster further research of AIMLR.
References
 Complex query answering with neural link predictors. In ICLR, Cited by: §2.2, §3.2.2.
 Dbpedia: a nucleus for a web of open data. In The semantic web, pp. 722–735. Cited by: §1.
 Introduction to description logic. Cambridge University Press. Cited by: §3.2.2.
 Description logics. Foundations of Artificial Intelligence 3, pp. 135–179. Cited by: §3.2.1.
 Appendix: description logic terminology. The Description logic handbook: Theory, implementation, and applications, pp. 485–495. Cited by: §3.1.2, §3.2.1.
 Translating embeddings for modeling multirelational data. NeurIPS 26. Cited by: §1.
 OWL2Vec*: embedding of owl ontologies. Machine Learning, pp. 1–33. Cited by: §2.3.
 Fuzzy logic based logical query answering on knowledge graph. arXiv preprint arXiv:2108.02390. Cited by: §2.2.
 Selfsupervised hyperboloid representations from logical queries over knowledge graphs. In WWW, pp. 1373–1384. Cited by: §2.1.
 Dictionary of philosophical logic. Edinburgh University Press. Cited by: §1.
 A case for abductive reasoning over ontologies.. In OWLED, Vol. 216. Cited by: §1.

A new metric for probability distributions
. IEEE Transactions on Information theory 49 (7), pp. 1858–1860. Cited by: §3.3.1.  Understanding the difficulty of training deep feedforward neural networks. In AISTATS, pp. 249–256. Cited by: §3.2.1, §3.2.2.
 A translation approach to portable ontology specifications. Knowledge acquisition 5 (2), pp. 199–220. Cited by: §1.
 Embedding logical queries on knowledge graphs. In NeurIPS, pp. 2030–2041. Cited by: §A.1, §1, §1, §2.1, §3.3.2, Table 2, §4.1.1, §4.1.1, Table 3.
 Universal representation learning of knowledge bases by jointly embedding instances and ontological concepts. In KDD, pp. 1709–1719. Cited by: §2.3.
 Biojoie: joint representation learning of biological knowledge bases. In Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, pp. 1–10. Cited by: §2.3.
 Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §4.1.3.
 Triangular norms. position paper i: basic analytical and algebraic properties. Fuzzy sets and systems 143 (1), pp. 5–26. Cited by: §3.2.2.
 Fuzzy sets and fuzzy logic. Vol. 4, Prentice hall New Jersey. Cited by: §1, §2.2, §3.2.1, §3.2.2.
 EL embeddings: geometric construction of models for the description logic el++. Cited by: §2.3.
 Semantic similarity and machine learning with ontologies. Briefings in bioinformatics 22 (4), pp. bbaa199. Cited by: §2.3.
 Fuzzy logic and the resolution principle. Journal of the ACM (JACM) 19 (1), pp. 109–119. Cited by: §3.2.1.
 Knowledge diffusion for neural dialogue generation. In ACL, pp. 1489–1498. Cited by: §1.
 EmEL++: embeddings for description logic. Cited by: §2.3.

Query2box: reasoning over knowledge graphs in vector space using box embeddings
. In ICLR, Cited by: §1, §1, §2.1, Table 2, §4.1.1, §4.1.1, §4.1.1, §4.1.3, Table 3.  Beta embeddings for multihop logical reasoning in knowledge graphs. NeurIPS 33. Cited by: §1, §1, §2.1, Table 2, §4.1.1, §4.1.1, §4.1.2, Table 3.
 Crispset qualitative comparative analysis (csqca). Configurational comparative methods: Qualitative comparative analysis (QCA) and related techniques 51, pp. 33–68. Cited by: §3.2.2.
 Opa2vec: combining formal and informal content of biomedical ontologies to improve similaritybased prediction. Bioinformatics 35 (12), pp. 2133–2140. Cited by: §2.3.
 Yago 4: a reasonable knowledge base. In European Semantic Web Conference, pp. 583–596. Cited by: §1.
 Entity alignment between knowledge graphs using attribute embeddings. In AAAI, Vol. 33, pp. 297–304. Cited by: §1.
 Analyzing differentiable fuzzy implications. In KR, pp. 893–903. Cited by: §3.2.2.
 The problem of induction. Stanford encyclopedia of philosophy. Cited by: §3.1.1.
 Improving conversational recommender systems via knowledge graph based semantic fusion. In KDD, pp. 1006–1014. Cited by: §1.
Appendix A Supplementary Materials
a.1. Inductive Reasoning in ABIN
Here, we elaborate the method to obtain the query embedding for inductive reasoning, i.e., with parameters . We use the integrated implementation^{5}^{5}5https://github.com/snapstanford/KGReasoning of GQE (Hamilton et al., 2018) to obtain . Specifically, the projection operation that project an entity or query embedding with relation is resolved by:
(18) 
where is another query embedding that is obtained in advance or an entity embedding obtained by looking up by rows. The intersection of two query embeddings and is resolved by:
(19) 
where denotes matrix concatenation over the last dimension, denote s the parameters of , and is a twolayer feedforward network with activation. and represent the first and second attention weights, respectively. The union of two query embeddings and is resolved by:
(20) 
where denotes the max operation over the last dimension.
Abduction  1p  2p  3p  2i  3i  pi  ip  2u  up  avg 

w/o CC  61.4  73.8  71.8  22.9  20.1  28.8  21.6  67.5  62.6  47.8 
w/o EC  58.9  69.7  68.0  42.8  39.6  41.2  22.0  66.4  62.9  52.4 
ABIN  62.4  83.7  83.6  50.9  43.7  45.0  29.9  67.2  67.3  59.3 
Induction  1p  2p  3p  2i  3i  pi  ip  2u  up  avg 
w/o CC  20.5  22.8  23.0  18.8  20.6  14.7  25.6  10.4  18.4  19.4 
w/o EC  19.4  19.1  20.5  19.1  20.1  15.0  25.7  9.1  20.3  18.7 
ABIN  34.6  28.0  29.0  44.6  54.8  21.4  40.6  17.8  23.0  32.6 
Algorithm 1 The learning procedure of ABIN.