Completing and Debugging Ontologies: state of the art and challenges

08/08/2019 ∙ by Patrick Lambrix, et al. ∙ 0

As semantically-enabled applications require high-quality ontologies, developing and maintaining as correct and complete as possible ontologies is an important, although difficult task in ontology engineering. A key step is ontology debugging and completion. In general, there are two steps: detecting defects and repairing defects. In this paper we formalize the repairing step as an abduction problem and situate the state of the art with respect to this framework. We show that there still are many open research problems and show opportunities for further work and advancing the field.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

In recent years many ontologies have been developed. Intuitively, ontologies can be seen as defining the basic terms and relations of a domain of interest, as well as the rules for combining these terms and relations. They are a key technology for the Semantic Web. The benefits of using ontologies include reuse, sharing and portability of knowledge across platforms, and improved documentation, maintenance, and reliability. Ontologies lead to a better understanding of a field and to more effective and efficient handling of information in that field.

Developing ontologies (see [1] for a survey on ontology libraries) and their alignments is not an easy task and often the resulting ontologies are not consistent or complete. Such ontologies, although often useful, also lead to problems when used in semantically-enabled applications. Wrong conclusions may be derived or valid conclusions may be missed. Semantically-enabled applications require high-quality ontologies and mappings. A key step towards this is completing and debugging, i.e., detecting and repairing defects in the ontologies and their alignments.

Defects in ontologies can take different forms (e.g., [2]). Syntactic defects are usually easy to find and to resolve. Defects regarding style include such things as unintended redundancy. More interesting and severe defects are the modeling defects such as missing is-a relations, and semantic defects such as unsatisfiable concepts and inconsistent ontologies. Modeling defects usually require domain experts to detect and repair them.

For instance, as an example of modeling defects, in [3] it was shown that for the two ontologies used in the Anatomy track in the Ontology Alignment Evaluation Initiative (OAEI, yearly event for evaluation of ontology alignment systems), at least 121 and 83, respectively, is-a relations that are correct in the domain are missing in these ontologies. It is also known that people that are not expert in knowledge representation often misuse and confuse equivalence, is-a and part-of (e.g. [4]). Regarding semantic defects, in [2], e.g., it was shown that the TAMBIS ontology contained 144 unsatisfiable concepts.

Defects in ontologies clearly influence the correctness of the intended modeling of a domain. Further, defects can have a large influence on the performance of semantically-enabled applications that use the ontologies. For instance, in ontology-based search, queries are refined and expanded by moving up and down the hierarchy of concepts. Incomplete structure in ontologies influences the quality of the search results. As an example, suppose we want to find articles in the MeSH (Medical Subject Headings,, controlled vocabulary of the National Library of Medicine, US) Database of PubMed ( using the term Scleral Disease in MeSH. By default the query (performed on March 4, 2019) will follow the hierarchy of MeSH and include more specific terms for searching, such as Scleritis. In this example run we retrieved 4615 documents. However, if the relation between Scleral Disease and Scleritis is missing in MeSH, we will miss 609 articles in the search result. (In earlier work, e.g., [5], the missed articles amounted to up to 55% of the original result.)

As more and more new ontologies were being developed, many with overlapping information, multiple ontologies were used in semantically-enabled applications. This, however, led to the need to detect the overlapping information in ontologies and mappings between the ontologies were created. A set of mappings between ontologies is called an alignment. These mappings are needed for a number of different tasks such as data integration, data warehousing, query answering, web service retrieval and B2B applications. An ontology network is a set of ontologies and alignments between these ontologies. Many ontology alignment systems that produce alignments, have been developed. As more and more ontologies are aligned, more and more mappings become available, and a number of systems and portals have been set up that store these (e.g., BioPortal (, Unified Medical Language System ( However, for all but the smallest ontologies, the mappings generated by tools and even by domain experts, contain mistakes and/or are not complete. For instance, BioPortal contains mappings for the OAEI Anatomy track with ca 20% of the mappings not correct and ca 20% of the correct mappings missing.

The study in [6] collected data using interviews and questionnaires from 148 ontology engineering projects from industry and academia in different areas such as information systems, commerce, multimedia and tourism. One of the findings was that quality of the developed ontologies is a major concern. Most projects did minor testing but tools for aiding in ontology evaluation may result in major efficiency gains. Further, one of the few ontology development methodologies that explicitly introduces ontology debugging and alignment is an extension of the eXtreme Design Methodology [7].

In this paper we review approaches for raising the quality of ontologies by repairing them. In general, completing and debugging requires two steps. In the detection step, for which much work exists, defects are found using different approaches including inspection, ontology learning or evolution (e.g., [8, 9]), using linguistic and logical patterns (e.g., [10, 11, 12, 13, 14, 15, 16]), by using knowledge intrinsic to an ontology network (e.g., [17, 18, 19, 20]

), or by using machine learning and statistical methods (e.g.,

[21, 8, 22, 23]).

In the repairing step the detected wrong information is removed and missing information is added. In this paper we focus on the repairing step which can be formalized as an abduction problem and for which there still are many open research problems. In Sect. 2 we formalize ontology repair (completion and debugging) as an abductive reasoning problem and introduce different preference relations between solutions that are relevant to this problem. In Sect. 3 and 4 we discuss the state of the art based on our formalization for ontologies and ontology networks, respectively. We discuss debugging, completion and the combination of these. Finally, we give some open problems in Sect. 5.

2 Ontology Repair

In this section we focus on repairing ontologies represented in description logics, where we have already detected wrong and missing information. We only discuss ontologies at concept level (sometimes called ’schema’ level) and not populated ontologies. We define this problem as an abductive reasoning problem and, as a repairing problem can have many solutions, we discuss preference relations between these.

2.1 Formalization

2.1.1 Repair

Definition 1

(Repair) Let be a TBox and be the set of all atomic concepts in . Let and be finite sets of TBox axioms. Let be an oracle that given a TBox axiom returns true or false. A repair for Complete-Debug-Problem CDP is any pair of finite sets of TBox axioms such that
(i) p : (p) = true;
(ii) q : (q) = false;
(iii) is consistent;
(iv) m : m;
(v) w : w.

Def. 1 formalizes the repair of an ontology for which missing and wrong information is given. An ontology is represented by a TBox with its set of atomic concepts . The identified missing and wrong information is represented by a set of missing axioms, and a set of wrong axioms. To repair the TBox, a set of axioms that are correct according to the oracle should be added to the TBox and a set of axioms that are not correct according to the oracle should be removed from the TBox such that the new TBox is consistent, the missing axioms are derivable from the new TBox and the wrong axioms are not derivable from the new TBox. As an example, consider the CDP in Fig. 1. Then , , , and are all repairs of the CDP.

T: {ax1: , ax2: , ax3: , ax4: , ax5: ,
       ax6: , ax7: , ax8: , ax9: ,
       ax10: }
C: {, , , , , , , }
Or(X) = true for X = ax2, ax3, ax4, ax5, ax7, ax8, ax9, ;
Or(X) = false for X = ax1, ax6, ax10, ,

M = {},
W = {, }

=({, }, {ax1, ax6, ax10}),
=({, }, {ax1, ax10}),
=({}, {ax1, ax10}),
=({}, {ax1, ax6}),
=({, }, {ax1, ax6})

Figure 1: Example complete-debug problem.

In general, the set of all axioms that are correct according to the domain and the set of all axioms that are not correct according to the domain are not known beforehand. Indeed, if these sets were given then we would only have to add the axioms of the first set to the TBox and remove the axioms in the second set from the TBox. The common case, however, is that we do not have these sets, but instead can rely on a domain expert that can decide whether an axiom is correct according to the domain or not. Therefore, in the formalization we introduce an oracle that represents the domain expert and that when given an axiom, returns true or false.

2.1.2 Influence of the quality of the oracle.

For we identified the following interesting cases. The first case is the all-knowing oracle. In this case the oracle’s answer is always correct. This is the ideal case, but may not always be achievable. Most current work considers this kind of oracle. In the second case, the limited all-knowing oracle, if answers, then the answer is correct, but it may not know the answer to all questions. This case represents a domain expert who knows a part of the domain well. An approximation of this case is when there are several domain experts who may have different opinions and we use a skeptical approach. Only if all domain experts give the same answer regarding the correctness of an axiom, do we consider the answer. In the third case can make mistakes regarding the validation of axioms. Axioms that are not correct according to the domain may be validated as correct and vice versa. This is the most common case. Although most current work assumes an all-knowing oracle, recent work used, in addition to an all-knowing oracle, also oracles with specific error rates in the evaluation of ontology alignment systems [24]. A lesson learned was that oracles with error rates up to 30% were still beneficial for the systems. The fourth case represents situations where no domain expert is available and there is no validation of axioms, such as in fully automated systems.

As noted, most current work considers an all-knowing oracle. With an all-knowing oracle we can check that p : Or(p) = true, and q : Or(q) = false and if this is not the case, we can remove the falsely identified defects. Therefore, we can, without loss of generality, assume that the axioms in really are missing, and the axioms in really are false. Further, regarding repairs, when using an all-knowing oracle, we know that all added axioms in are correct according to the domain and all removed axioms in are false according to the domain. Further, for an all-knowing oracle, we know that (and then = ). When using other oracles, we cannot be sure that the given missing and wrong axioms really are missing and wrong, respectively. Therefore, oracles that make mistakes or do not know the correctness of all axioms may start with wrong input. Also, wrong axioms may be added and correct axioms may be removed during the repairing. These issues may have a negative effect on the quality of the repaired ontology.

In practice, when using domain experts, it is not possible to know which kind of domain expert is used. When only one domain expert is available it is reasonable for the systems to assume that an all-knowing expert is used, although we should be aware that mistakes can occur. The more domain experts are available, a skeptical approach or a voting approach may be used for raising the quality of the ontology.

2.2 Preference relations

As there may exist many possible repairs for a given CDP, and not all are equally interesting, it is necessary to define preference relations between repairs.

2.2.1 Basic preferences

From the completion perspective of a complete-debug-problem it is important to find repairs that add to the ontology as much information as possible that is correct according to the domain, while from the correctness perspective as much wrong information as possible should be removed. Def. 2 and 3 formalize these intuitions, respectively. Further, Def. 4 defines a classical preference relation for abduction problems related to removing redundancy.

Definition 2

(more complete) Let and be two repairs for CDP. is more complete than (or is preferred to with respect to ’more complete’) iff
and are equally complete iff

Definition 3

(less incorrect) Let and be two repairs for CDP. is less incorrect than (or is preferred to with respect to ’less incorrect’) iff
and are equally incorrect iff

Definition 4

(subset) Let and be two repairs for CDP.
iff .
iff .
iff .
(If , we also say that is preferred to with respect to .)

As examples, for the CDP in Fig. 1 we have that and are equally complete, but is less incorrect than . Further, and are equally incorrect and complete, and is less incorrect and more complete than .

2.2.2 Preferred repairs with respect to a basic preference

Based on these preference relations we can define repairs that are preferred with respect to one particular preference relation (Def. 5 - 7).

Definition 5

(maximally complete) A repair for CDP is said to be maximally complete (or preferred with respect to ’more complete’) iff there is no repair which is more complete than .

Definition 6

(minimally incorrect) A repair for CDP is said to be minimally incorrect (or preferred with respect to ’less incorrect’) iff there is no repair which is less incorrect than .

Definition 7

(subset minimal) A repair for CDP is said to be subset minimal (or preferred with respect to ) iff there is no repair such that and .

As examples, for the CDP in Fig. 1 we have that and are subset minimal. Depending on the validity of axioms not shown in Fig. 1, , , and may be maximally complete while may be minimally incorrect.

2.2.3 Combining preferences

In practice, however, all of the criteria regarding completeness, correctness and redundancy are desirable. Therefore, we define different ways to combine these criteria. First, we need to define when a repair dominates another repair with respect to preference relations (Def. 8). Then we define the combination of preferences with priority to one of the preference relations (Def. 9) and with equal priority for the preference relations (Def. 10).

Definition 8

(dominate) Let and be two repairs for CDP. dominates with respect to a set of preference relations {more complete, less incorrect, } if is more than or equally preferred to for all preference relations in is more preferred to for at least one of the preference relations in .

Definition 9

(combining with priority to one of the preference relations) Let {more complete, less incorrect, }. Let {more complete, less incorrect, } {}. A repair for CDP is said to be X-optimal with respect to iff is preferred with respect to and there is no other repair that is preferred with respect to and dominates with respect to .

Definition 10

(combining with equal priority) A repair for CDP is said to be skyline-optimal with respect to iff there is no other repair that dominates with respect to .

We note that if a repair is X-optimal with respect to , then it is skyline-optimal with respect to {X}.

As examples, for the CDP in Fig. 1 we have that and are -optimal with respect to {less incorrect}. Depending on the validity of axioms not shown in Fig. 1, may be -optimal with respect to {more complete} and can be less-incorrect-optimal with respect to {more complete} and {more complete, } and more-complete-optimal with respect to {less incorrect}.

The advantage of maximally complete and more-complete-optimal repairs is that a maximal body of correct information is added to the ontology and for the latter without redundancy and with removing as much wrong information as possible. The advantage of minimally incorrect and less-incorrect-optimal repairs is that a maximal body of wrong information is removed from the ontology and for the latter without redundancy and with adding as much correct information as possible. Although these are the most attractive repairs, in practice it is not clear how to generate such repairs, apart from a usually infeasible brute-force procedure that checks the correctness of all axioms with the oracle. (Although a strategy can be devised to check all without asking the oracle for each axiom, the number of requests will still be large.) Repairs prioritizing subset minimality ensure that there is no redundancy. The advantage of removing redundant axioms is the reduction of computation time as well as the reduction of unnecessary user interaction. However, in some cases redundancy may be interesting. For instance, developers may want to have explicitly stated axioms in the ontologies even though they are redundant. This can happen, for instance, for efficiency reasons in applications or as domain experts have validated asserted axioms, these may be considered more trusted than derived axioms. Further, focusing on redundancy may lead to less complete or more incorrect repairs. Skyline-optimal is a relaxed criterion. When, for instance, = {more complete, less incorrect}, then a skyline-optimal repair with respect to is a preferred repair with respect to correctness for a certain level of completeness, or a preferred repair with respect to completeness for a certain level of correctness. In practice, as it is not clear how to generate more-complete-optimal and less-incorrect-optimal repairs, a skyline-optimal repair may be the next best thing and, in some cases (e.g., Sect. 3.2) it is easy to generate a skyline-optimal repair. However, in general, the difficulty lays in reaching as high levels of completeness and as low levels of incorrectness as possible.

3 State of the art- ontologies

Most of the current work has focused on the correctness or the completeness of ontologies, but very few work has dealt with both. However, a naive combination of a completion step and a debugging step does not necessarily lead to repairs for the combined problem. In this section we discuss current work.

3.1 Correctness

When only dealing with repairing the inconsistency or incoherence of Tboxes (semantic defects), only wrong information is dealt with. Therefore, in Def. 1, and . In most current approaches the domain expert is not included. This means that choices are made solely based on the logic and that correct axioms may be removed from the ontologies. Therefore, not all solutions may actually be repairs as defined in Def. 1 as requirement (ii) may not be satisfied.

There is much work on repairing semantic defects. Most approaches are based on finding explanations or justifications for the defects using a glass-box or black-box approach [2]. A glass-box approach is based on the internals of the reasoning algorithm of a description logic reasoner. A black-box approach uses a description logic reasoner as an oracle to determine answers to standard description logic reasoning tasks such as concept satisfiability or subsumption with respect to an ontology.

A general approach for repairing incoherent ontologies is the following (adapted from [25]). (For inconsistent ontologies we can use a similar approach.) For a given set of unsatisfiable concepts for an ontology, compute the minimal explanations for the defects, i.e., the minimal reasons for the unsatisfiability of concepts. These minimal reasons for the unsatisfiability of a concept are sets of axioms and are called minimal unsatisfiability-preserving sub-TBoxes (MUPS) or justifications for the unsatisfiability. We need to compute these MUPS or justifications for all unsatisfiable concepts. From these we can compute the minimal incoherence-preserving sub-TBoxes (MIPS) which are the smallest sets of axioms in the original Tbox that cause that TBox to be incoherent. To repair the incoherent TBox, we need to remove at least one axiom from each MIPS. We now define the notions in this general repairing approach formally.

The definition of MUPS is given in Def. 11. A MUPS in a consistent TBox can be seen as a justification (Def. 12) for an unsatisfiable concept. Indeed, if we instantiate in Def. 12 with we obtain the MUPS for . The definition of MIPS is given in Def. 13.

Definition 11

(MUPS) [25] Let be a TBox and be an unsatisfiable concept in . A set of axioms is a minimal unsatisfiability-preserving sub-TBox (MUPS) if is unsatisfiable in and is satisfiable in every sub-TBox .

Definition 12

(Justification) (similar to [26]) Let be a consistent TBox and . A set of axioms is a justification for in if and

Definition 13

(MIPS) [25] Let be an incoherent TBox. A TBox is a minimal incoherence-preserving sub-TBox (MIPS) if is incoherent and every sub-TBox is coherent.

As mentioned, to repair the incoherent TBox, we need to remove at least one axiom from each MIPS. Essentially, this means we should compute a hitting set (Def. 14) of the set of MIPS and remove the hitting set from the TBox. In [25] these hitting sets are called pinpoints. Complexity results regarding this problem are given in [27, 28, 29].

Definition 14

(hitting set) ([30]) Let be a collection of sets. A hitting set for is a set such that .

As an example, consider the TBox in Fig. 1. This TBox is incoherent with unsatisfiable concepts and . The set of MUPSs for is {{ax1, ax3, ax4}, {ax2, ax6, ax7, ax9, ax10}} while the set of MUPSs for is {{ax6, ax7, ax9, ax10}}. The set of MIPSs is {{ax1, ax3, ax4}, {ax6, ax7, ax9, ax10}}. A possible hitting set is {ax1, ax6}.

In general, there may be several hitting sets for the set of MIPS. Different approaches use different heuristics for ranking the possible repairs.

The first tableau-based algorithm for debugging of an ontology was proposed in [31, 25]. (For an overview of how a tableau-based reasoner works, see, e.g., [32].) The work was motivated by the development of the DICE (Diagnoses for Intensive Care Evaluation) terminology. A glass-box approach was used for an reasoner. The branches in the tableau-based reasoner were used to compute MUPS. The MIPS were computed by taking a subset-reduction of the union of all MUPSs, where the subset-reduction of a set S of TBoxes is the smallest sub-set of S such that for all TBoxes T in S there is a TBox T’ in the subset-reduction that is a subset of T [31]. Computing MUPS and MIPS for an unfoldable TBox was shown to be in PSPACE. Computing hitting sets takes linear time for the non-minimal case while the problem is NP-complete for the minimal case [25]. This approach was implemented for unfoldable TBoxes in the system MUPSter [33]. The tableau algorithm in [34] can be seen as an extension of this work. It computes maximally satisfiable sub-TBoxes and does not need individual steps for computing MUPS and applying the hitting set algorithm. The DION system [33] uses a bottom-up algorithm to compute MUPS. It is based on for an unsatisfiable concept P finding two sets of axioms and such that P is satisfiable in , but not in . Then subsets of are computed such that P is unsatisfiable in . By removing redundancy from these sets we obtain MUPS. For efficiency reasons not all sets of axioms are checked, but the search is guided by a relevance function, e.g., by using only axioms that are in some way relevant to the unsatisfiable concept. In [2] a glass-box technique is used to compute MUPS (called set of support in [2]) for OWL ontologies (). In [26] a method was proposed to calculate all justifications of an unsatisfiable concept. Both a glass-box and black-box technique are presented for computing a single justification. The glass-box technique is an extension from [2], while the black-box technique is based on an expansion stage where axioms are added to an initially empty set until a concept becomes unsatisfiable and a shrinking step where extraneous axioms are removed. Then given an initial justification, a black-box method computes all justifications using a variation of the hitting set tree algorithm [30].

As there may be different ways to repair the ontologies and as computing justifications can be expensive, different heuristics and optimization approaches have been proposed (e.g., [35]). In [25] a heuristic is used stating that axioms appearing in more MIPSs are likely to be more erroneous. Therefore, axioms appearing in the most MIPSs are removed first (or, in other words, are first added to the hitting set). In [36] an arity-based heuristic is used which is similar to the heuristic in [25]. Further, [36] introduces heuristics based on the impact on the ontology when an axiom is removed and based on test cases applied by a user, e.g., by specifying desired and undesired entailments, which may be seen as an oracle that has validated certain entailments a priori. They also propose using provenance information about the axioms as well as the syntactic use of the elements in the axioms in the other axioms in the ontology to rank the axioms. Reiter’s hitting set algorithm is modified to take the axiom rankings into account. In [2] root concepts are repaired first. A root concept is an unsatisfiable concept for which a contradiction in its definition does not depend on the unsatisfiability of another concept. The other unsatisfiable concepts are then derived concepts. Repairing root concepts may automatically repair derived concepts. In [37] a notion of relevance between axioms is defined and used to guide the computation of justifications. Patterns explaining unsatisfiability are used in [38] to optimize finding MUPS. In [39] it is possible to require that certain axioms are derivable from the repaired ontology. Further, queries are generated regarding the correctness of axioms and based on the answer of an oracle repairs will be accepted or rejected.

An approach that has not been proposed earlier, but that follows naturally from the definitions and preferences of Sect. 2, is to use the oracle for the axioms in the MIPSs. For every MIPS, remove the axioms such that (ax) = true. If at least one of the MIPS becomes the empty set, then there is no repair unless we are willing to remove correct information. Assuming we have non-empty MIPSs after the removal of correct axioms, a hitting set would result in a repair. When redundancy is removed, we obtain a subset minimal repair. Another possibility, as we have checked the correctness using the oracle, is to use all remaining axioms in all MIPSs (as for these axioms we have that (ax) = false). This repair is less incorrect than the repairs obtained using hitting sets.

There are also approaches that map the debugging problem into a revision problem (e.g., [40, 41]). A revision state [40] is a tuple of ontologies (, , ) where , , and = . represents the wanted consequences of , while represents the unwanted consequences. In a complete revision state we also have that = . For a CDP, could be initialized with (and when dealing with completion, could be initialized with ). The approach in [40] is an interactive method where questions are asked to an oracle to decide whether axiom is correct or not, and then consequences are computed and revision states are updated iteratively. The decision on which questions to ask are based on the computation of an axiom impact measure. In [41] a MIPS approach is used in the definition of the revision operator.

3.2 Completeness

Most of the work on completing ontologies has dealt with completing the is-a structure of ontologies. An all-knowing oracle is often assumed. Therefore, in Def. 1, p : (p) = true, and .

There is not much work on the repairing of missing is-a structure. Most approaches just add the detected missing is-a relations. This conforms to the solution where = . When is consistent and p : (p) = true, we are guaranteed that is a solution. In the case all missing is-a relations were detected in the detection phase, this is essentially all that can to be done (except for removing redundancy, if so desired). If not all missing is-a relations were detected - and this is the common case - there are different ways to repair the ontology which are not all equally interesting and we can use the earlier defined preference relations.

As these approaches do not deal with correctness, Def. 3 and 6 are not used, and should be removed in Def. 4. In Def. 8 and 10, = {more complete, }. In Def. 9, ’less incorrect’ should be removed. In this case, the semantically maximal solutions in [42] are a special case of the maximally complete repairs where only subsumption axioms between atomic concepts are used. Further, the X-optimal and skyline-optimal repairs combine only completeness and subset minimality.

Interactive solutions to this completion problem have been proposed for taxonomies [3, 43, 44], for TBoxes [42, 44] and for TBoxes [45]. All algorithms compute logically correct solutions which then need to be validated for correctness according to the domain by a domain expert. It is assumed that the axioms and represent subsumption between atomic concepts in the ontologies. The algorithms for taxonomies and (normalized) TBoxes (unified notation in [44]) require that is consistent and m : Or(m) = true, and thus is a repair. The algorithms start with a first step that computes skyline-optimal repairs with respect to { more complete, } for each missing is-a relation. This step is different for different representation languages of the TBox. For taxonomies the algorithm tries to find ways to repair a missing is-a relation by adding axioms of the form where and . For , additionally, is-a relations of the form are repaired by repairing . For also role hierarchies and role inclusions need to be taken into account. Then the algorithms combine and modify these repairs into a single skyline-optimal repair for the whole set of missing is-a relations. Further, the algorithms repeat this process iteratively by solving new completion problems where the new is set to the added axioms in in the previous iteration. The union of the sets of added axioms of all iterations (with optionally removal of redundancy) is the final repair. It is shown that the skyline-optimal repairs (including the final repair if redundancy is removed) found during the iterations of the new completion problems are skyline-optimal repairs for the original completion problem that are equally or more complete than the repairs found in the first iteration. Complexity results for the existence problem (does a repair exist?), relevance problem (does a repair containing a given axiom exist?) and necessity problem (do all repairs contain a given axiom?) in general and with respect to different preferences are given for and in [42, 44]. In [45] an approach is proposed for TBoxes by modifying a tableau-based reasoner. Repairs are found by closing leaf nodes in the completion graphs generated by trying to disprove missing is-a relations using the tableau reasoner. Open leaf nodes are closed by finding pairs of statements of the form and and asserting then that . Additionally, the same technique as for taxonomies is applied.

A non-interactive solution, i.e., without validation of an oracle, that is independent of the constructors of the description logic (e.g., tested with ontologies with expressivity up to ) is proposed in [46]. In contrast to the previous approaches where the repairs only contain subsumption axioms between existing concepts, this approach introduces justification patterns that can be instantiated with existing concepts or ’fresh’ concepts. Further, the notion of justification pattern-based repairs is introduced which are a kind of repairs that are subset-minimal. Methods for computing all justification patterns as well as justification-based repairs are given.

3.3 Completeness and correctness

There is very little work on dealing with both completeness and correctness. In [20, 19] two versions of the RepOSE system are presented that allow for debugging and completing the is-a structure of ontologies (and mappings between ontologies) in an iterative and interleaving way. Wrong information is removed by calculating justifications and allowing a user to mark wrong is-a relations. Missing information is added using the techniques in Sect. 3.2. As the system always warns the user of influences of new additions or deletions on previous changes, the system can guarantee a repair if such exists, but it does not always guarantee a skyline-optimal solution.

4 State of the art - ontology networks

An ontology network is a collection of ontologies and pairwise alignments between these ontologies. Completing and debugging of such ontology networks has received more and more attention. Similar to single ontologies, also for networks the quality is dependent on the availability of domain experts and completely automatic systems may reduce the quality [47].

Our definitions in Sect. 2 and 3 can be used for ontology networks by creating a TBox from the network (i.e., it includes all axioms of all TBoxes from the ontologies and treats all mappings in all alignments in the network as axioms) and using this TBox in the definitions. It also follows that the techniques for single ontologies can be used for ontology networks. However, in much of the current research the axioms in the ontologies in the network and the axioms in the alignments are distinguished and treated differently.

The field of ontology alignment [48] deals with completeness of alignments (and thus only completion of the alignments, not of the ontologies in the networks). Many ontology alignment systems have been developed and overviews can be found in, e.g., [49, 50, 51, 48, 52, 53, 54] and at the ontology matching web site ( Usually ontology alignment systems take as input two source ontologies and output an alignment. Systems can contain a pre-processing component that, e.g., partition the ontologies into mappable parts thereby reducing the search space for finding mapping suggestions. Further, a matching component uses matchers that calculate similarities between the entities from the different source ontologies or mappable parts of the ontologies. They often implement strategies based on linguistic matching, structure-based strategies, constraint-based approaches, instance-based strategies, strategies that use auxiliary information or a combination of these. Each matcher utilizes knowledge from one or multiple sources. Mapping suggestions are then determined by combining and filtering the results generated by one or more matchers. Common combination strategies are the weighted-sum and the maximum-based strategies. The most common filtering strategy is the threshold filtering. Many systems output the found mappings suggestions as an alignment. However, it is well-known that to improve the quality user validation is necessary and several systems allow for user interaction in the different steps of the alignment including validation. Some systems also introduce other components such as recommendation of the settings for the different components in the system, e.g., [55].

Regarding correctness, most approaches deal with mapping repair where mappings rendering the network incoherent or inconsistent are removed. Usually, the axioms in the ontologies are considered more trustworthy than the mappings and thus mappings are removed, rather than axioms in the ontologies. Although detection of defects can be different for different existing systems, justification-based techniques are often used for the repairing as in [56], and the Radon [57], ALCOMO [58], LogMap [59, 60] and AgreementMakerLight [61] systems. Additional heuristics than the ones in Sect. 3 could be used. For instance, the conservativity principle [62] states that the integrated ontology should not induce any change in the concept hierarchies of the input ontologies. In [58, 63, 61] the confidence values of the mappings are taken into account and in [56] a semantic similarity measure between concepts in the mappings is used.

Similar to the case of ontologies, some approaches for ontology networks use a revision approach, e.g., [64, 65, 66]. Usually, the ontologies remain the same, but the set of mappings is revised.

An approach that distinguishes between axioms in the ontologies and in the alignments, but gives equal priority to them using approaches in Sect. 3 is [19, 20].

5 Opportunities

In this paper we have defined a framework for completing and debugging ontologies and shown the state of the art in the field. It is clear that many research opportunities still exist.

5.1 Within the framework

Many approaches have been proposed regarding correctness, but finding (preferred) repairs in an acceptable time is still an issue. Further, few approaches make use of an oracle.

There is relatively few work on dealing with completeness. The system that allows for user interaction deals with light-weight ontologies, while the work that allows for higher expressivity is non-interactive.

Even fewer work deals with completion and correctness. We need work on algorithms guaranteeing different kinds of preferred repairs, for instance, to find skyline-optimal solutions with as high levels of completeness and as low levels of incorrectness as possible.

There is also a need for complexity results of the completion-debugging problem, and for optimization techniques and heuristics.

Further, most work deals with all-knowing oracles and ways to deal with other kinds of oracles are needed.

Also, one of the most needed contributions is the development of tools.

5.2 Extensions based on current work

The formalization of the problem may be extended by using the notion of background knowledge which represents parts of the ontology that are asserted to be correct and therefore should not be changed [39].

Another extension is axiom weakening [67, 68, 69] where for debugging, instead of removing axioms, some axioms may be weakened, e.g., an equivalence axiom becomes a subsumption axiom.

Further, some approaches deal with populated ontologies and use the instances in detection or repairing, e.g., [70, 71, 72].


This work has been financially supported by the Swedish e-Science Research Centre (SeRC), the Swedish Research Council (Vetenskapsrådet, dnr 2018-04147) and the Horizon 2020 project SPIRIT (grant agreement No 786993).


  • [1] M. d’Aquin, N. F. Noy, Where to Publish and Find Ontologies? A Survey of Ontology Libraries, Journal of Web Semantics 11 (2012) 96–111. doi:10.1016/j.websem.2011.08.005.
  • [2] A. Kalyanpur, B. Parsia, E. Sirin, J. Hendler, Debugging Unsatisfiable Classes in OWL Ontologies, Journal of Web Semantics 3 (4) (2005) 268–293. doi:10.1016/j.websem.2005.09.005.
  • [3] P. Lambrix, Q. Liu, H. Tan, Repairing the missing is-a structure of ontologies, in: A. Gomez-Perez, Y. Yu, Y. Ding (Eds.), The Semantic Web, Fourth Asian Conference, ASWC 2009, Shanghai, China, December 6-9, 2009. Proceedings, Vol. 5926 of Lecture Notes in Computer Science, Springer, 2009, pp. 76–90. doi:10.1007/978-3-642-10871-6_6.
  • [4] C. Conroy, R. Brennan, D. O’Sullivan, D. Lewis, User Evaluation Study of a Tagging Approach to Semantic Mapping, in: L. Aroyo, P. Traverso, F. Ciravegna, P. Cimiano, , T. Heath, E. Hyvönen, R. Mizoguchi, E. Oren, M. Sabou, E. Simperl (Eds.), The Semantic Web: Research and Applications, 6th European Semantic Web Conference, ESWC 2009 Heraklion, Crete, Greece, May 31-June 4, 2009 Proceedings, Vol. 5554 of Lecture Notes in Computer Science, Springer, 2009, pp. 623–637. doi:10.1007/978-3-642-02121-3_46.
  • [5] Q. Liu, P. Lambrix, A System for Debugging Missing Is-a Structure in Networked Ontologies, in: P. Lambrix, G. Kemp (Eds.), Data Integration in the Life Sciences, 7th International Conference, DILS 2010, Gothenburg, Sweden, August 25-27, 2010. Proceedings, Vol. 6254 of Lecture Notes in Computer Science, Springer, 2010, pp. 50–57. doi:10.1007/978-3-642-15120-0_5.
  • [6] E. Simperl, M. Mochol, T. Bürger, Achieving maturity: the state of practice in ontology engineering in 2009, International Journal of Computer Science and Applications 7(1) (2010) 45–65.
  • [7] Z. Dragisic, P. Lambrix, E. Blomqvist, Integrating ontology debugging and matching into the extreme design methodology, in: E. Blomqvist, P. Hitzler, A. Krisnadhi, T. Narock, M. Solanki (Eds.), Proceedings of the 6th Workshop on Ontology and Semantic Web Patterns, Vol. 1461 of CEUR Workshop Proceedings, 2015.
  • [8] A. Maedche, V. Pekar, S. Staab, Ontology learning part one - on discovering taxonomic relations from the web, in: Zhong, Liu, Yao (Eds.), Web Intelligence, Springer, 2003, pp. 301–320. doi:10.1007/978-3-662-05320-1_14.
  • [9] P. Buitelaar, P. Cimiano, , B. Magnini, Ontology Learning from Text: Methods, Evaluation and Applications, IOS Press, 2005.
  • [10] M. A. Hearst, Automatic Acquisition of Hyponyms from Large Text Corpora, in: Proceedings of the 14th International Conference on Computational Linguistics, Vol. 2, 1992, pp. 539–545. doi:10.3115/992133.992154.
  • [11] W. R. van Hage, S. Katrenko, G. Schreiber, A method to combine linguistic ontology-mapping techniques, in: Y. Gil, E. Motta, V. R. Benjamins, M. A. Musen (Eds.), The Semantic Web - ISWC 2005, 4th International Semantic Web Conference, ISWC 2005, Galway, Ireland, November 6-10, 2005, Proceedings, 2005, pp. 732–744. doi:10.1007/11574620_52.
  • [12] P. Wang, B. Xu, Debugging ontology mappings: a static approach, Computing and Informatics 27 (2008) 21–36.
  • [13] O. Corcho, C. Roussey, L. M. Vilches Blázquez, I. Pérez, Pattern-based OWL ontology debugging guidelines, in: Workshop on Ontology Patterns, 2009, pp. 68–82.
  • [14] E. Beisswanger, U. Hahn, Towards valid and reusable reference alignments - ten basic quality checks for ontology alignments and their application to three different reference data sets, Journal of Biomedical Semantics 3 (Suppl 1) (2012) S4:1–S4:14. doi:10.1186/2041-1480-3-S1-S4.
  • [15]

    C. M. Keet, Detecting and revising flaws in OWL object property expressions, in: A. ten Teije, J. Völker, S. Handschuh, H. Stuckenschmidt, M. d’Aquin, A. Nikolov, N. Aussenac-Gilles, N. Hernandez (Eds.), Knowledge Engineering and Knowledge Management - 18th International Conference, EKAW 2012, Galway City, Ireland, October 8-12, 2012. Proceedings, 2012, pp. 252–266.

  • [16] M. Poveda-Villalón, A. Gómez-Pérez, M. C. Suárez-Figueroa, Oops! (ontology pitfall scanner!): An on-line tool for ontology evaluation, International Journal on Semantic Web and Information Systems 10 (2) (2014) 7–34. doi:10.4018/ijswis.2014040102.
  • [17] M. Bada, L. Hunter, Identification of OBO Nonalignments and Its Implications for OBO Enrichment, Bioinformatics 24 (12) (2008) 1448–1455. doi:10.1093/bioinformatics/btn194.
  • [18] P. Lambrix, Q. Liu, Using partial reference alignments to align ontologies, in: L. Aroyo, P. Traverso, F. Ciravegna, P. Cimiano, T. Heath, E. Hyvönen, R. Mizoguchi, E. Oren, M. Sabou, E. P. B. Simperl (Eds.), The Semantic Web: Research and Applications, 6th European Semantic Web Conference, ESWC 2009, Heraklion, Crete, Greece, May 31-June 4, 2009, Proceedings, 2009, pp. 188–202. doi:10.1007/978-3-642-02121-3_17.
  • [19] V. Ivanova, P. Lambrix, A unified approach for aligning taxonomies and debugging taxonomies and their alignments, in: P. Cimiano, Ó. Corcho, V. Presutti, L. Hollink, S. Rudolph (Eds.), The Semantic Web: Semantics and Big Data, 10th International Conference, ESWC 2013, Montpellier, France, May 26-30, 2013. Proceedings, 2013, pp. 1–15. doi:10.1007/978-3-642-38288-8_1.
  • [20] P. Lambrix, V. Ivanova, A unified approach for debugging is-a structure and mappings in networked taxonomies, Journal of Biomedical Semantics 4 (2013) 10. doi:10.1186/2041-1480-4-10.
  • [21] A. Maedche, S. Staab, Discovering conceptual relations from text

    , in: W. Horn (Ed.), ECAI 2000, Proceedings of the 14th European Conference on Artificial Intelligence, Berlin, Germany, August 20-25, 2000, 2000, pp. 321–325.

  • [22] E. Zavitsanos, G. Paliouras, G. A. Vouros, S. Petridis, Discovering subsumption hierarchies of ontology concepts from text corpora, in: IEEE / WIC / ACM International Conference on Web Intelligence, 2007, pp. 402–408. doi:10.1109/WI.2007.55.
  • [23] V. Spiliopoulos, G. A. Vouros, V. Karkaletsis, On the discovery of subsumption relations for the alignment of ontologies, Journal of Web Semantics 8 (2010) 69–88. doi:10.1016/j.websem.2010.01.001.
  • [24] Z. Dragisic, V. Ivanova, P. Lambrix, D. Faria, E. Jiménez-Ruiz, C. Pesquita, User validation in ontology alignment, in: P. Groth, E. Simperl, A. Gray, M. Sabou, M. Krötzsch, F. Lecue, F. Flöck, Y. Gil (Eds.), The Semantic Web - ISWC 2016, 15th International Semantic Web Conference, Kobe, Japan, October 17-21, 2016, Proceedings, Part I, Vol. 9981 of Lecture Notes in Computer Science, Springer, 2016, pp. 200–217. doi:10.1007/978-3-319-46523-4_13.
  • [25] S. Schlobach, Debugging and Semantic Clarification by Pinpointing, in: A. Gomez-Perez, J. Euzenat (Eds.), The Semantic Web: Research and Applications - Second European Semantic Web Conference, ESWC 2005, Heraklion, Crete, Greece, May 29-June 1, 2005. Proceedings, 2005, pp. 226–240. doi:10.1007/11431053_16.
  • [26] A. Kalyanpur, B. Parsia, M. Horridge, E. Sirin, Finding All Justifications of OWL DL Entailments, in: K. Aberer, K.-S. Choi, N. Noy, D. Allemang, K.-I. Lee, L. Nixon, J. Golbeck, P. Mika, D. Maynard, R. Mizoguchi, G. Schreiber, P. Cudre-Mauroux (Eds.), The Semantic Web - 6th International Semantic Web Conference, 2nd Asian Semantic Web Conference, ISWC 2007 + ASWC 2007, Busan, Korea, November 11-15, 2007. Proceedings, 2007, pp. 267–280. doi:10.1007/978-3-540-76298-0_20.
  • [27] R. Peñaloza, B. Sertkaya, On the complexity of axiom pinpointing in the EL family of description logics, in: F. Lin, U. Sattler, M. Truszczynski (Eds.), Principles of Knowledge Representation and Reasoning: Proceedings of the Twelfth International Conference, KR 2010, Toronto, Ontario, Canada, May 9-13, 2010, 2010.
  • [28] R. Peñaloza, B. Sertkaya, Complexity of axiom pinpointing in the dl-lite family of description logics, in: H. Coelho, R. Studer, M. J. Wooldridge (Eds.), ECAI 2010 - 19th European Conference on Artificial Intelligence, Lisbon, Portugal, August 16-20, 2010, Proceedings, 2010, pp. 29–34. doi:10.3233/978-1-60750-606-5-29.
  • [29] R. Peñaloza, B. Sertkaya, Understanding the complexity of axiom pinpointing in lightweight description logics, Artificial Intelligence 250 (2017) 80–104. doi:10.1016/j.artint.2017.06.002.
  • [30] R. Reiter, A theory of diagnosis from first principles, Artificial Intelligence 32(1) (1987) 57–95. doi:10.1016/0004-3702(87)90062-2.
  • [31] S. Schlobach, R. Cornet, Non-Standard Reasoning Services for the Debugging of Description Logic Terminologies, in: G. Gottlob, T. Walsh (Eds.), Proceedings of the 18th International Joint Conference on Artificial Intelligence, 2003, pp. 355–360.
  • [32] F. Baader, W. Nutt, Basic description logics, in: F. Baader, D. Calvanese, D. McGuinness, D. Nardi, P. Patel-Schneider (Eds.), The Description Logic Handbook, Cambridge University Press, 2003, Ch. 2, pp. 43–95.
  • [33]

    S. Schlobach, Z. Huang, R. Cornet, F. van Harmelen, Debugging Incoherent Terminologies, Journal of Automated Reasoning 39 (3) (2007) 317–349.

  • [34] T. Meyer, K. Lee, R. Booth, J. Pan, Finding maximally satisfiable terminologies for the description logic ALC, in: A. Cohn (Ed.), AAAI’06 - Proceedings of the 21st national conference on Artificial intelligence, AAAI Press, 2006, pp. 269–274.
  • [35] Q. Ji, Z. Gao, Z. Huang, M. Zhua, Measuring effectiveness of ontology debugging systems, Knowledge-Based Systems 71 (2014) 169–186. doi:10.1016/j.knosys.2014.07.023.
  • [36] A. Kalyanpur, B. Parsia, E. Sirin, B. Cuenca-Grau, Repairing Unsatisfiable Concepts in OWL Ontologies, in: Y. Sure, J. Domingue (Eds.), The Semantic Web: Research and Applications 3rd European Semantic Web Conference, ESWC 2006 Budva, Montenegro, June 11-14, 2006 Proceedings, 2006, pp. 170–184. doi:10.1007/11762256_15.
  • [37] Q. Ji, G. Qi, P. Haase, A relevance-directed algorithm for finding justifications of DL entailments, in: A. Gómez-Pérez, Y. Yu, Y. Ding (Eds.), The Semantic Web, Fourth Asian Conference, ASWC 2009, Shanghai, China, December 6-9, 2009. Proceedings, 2009, pp. 306–320. doi:10.1007/978-3-642-10871-6_21.
  • [38] Q. Ji, Z. Gao, Z. Huang, M. Zhu, An efficient approach to debugging ontologies based on patterns, in: J. Z. Pan, H. Chen, H. Kim, J. Li, Z. Wu, I. Horrocks, R. Mizoguchi, Z. Wu (Eds.), The Semantic Web - Joint International Semantic Technology Conference, JIST 2011, Hangzhou, China, December 4-7, 2011. Proceedings, 2011, pp. 425–433. doi:10.1007/978-3-642-29923-0_33.
  • [39] K. M. Shchekotykhin, G. Friedrich, P. Fleiss, P. Rodler, Interactive ontology debugging: Two query strategies for efficient fault localization, Journal of Web Semantics 12 (2012) 88–103. doi:10.1016/j.websem.2011.12.006.
  • [40] N. Nikitina, S. Rudolph, B. Glimm, Interactive ontology revision, Journal of Web Semantics 12 (2012) 118–130. doi:10.1016/j.websem.2011.12.002.
  • [41] X. Fu, G. Qi, Y. Zhang, Z. Zhou, Graph-based approaches to debugging and revision of terminologies in dl-lite, Knowledge-Based Systems 100 (2016) 1–12. doi:10.1016/j.knosys.2016.01.039.
  • [42] F. Wei-Kleiner, Z. Dragisic, P. Lambrix, Abduction framework for repairing incomplete el ontologies: Complexity results and algorithms, in: AAAI’14 - Proceedings of the 28st national conference on Artificial intelligence, 2014, pp. 1120–1127.
  • [43] P. Lambrix, Q. Liu, Debugging the missing is-a structure within taxonomies networked by partial reference alignments, Data and Knowledge Engineering 86 (2013) 179–205. doi:10.1016/j.datak.2013.03.003.
  • [44] P. Lambrix, F. Wei-Kleiner, Z. Dragisic, Completing the is-a structure in light-weight ontologies, Journal of Biomedical Semantics 6 (2015) 12:1–12:26. doi:10.1186/s13326-015-0002-8.
  • [45] P. Lambrix, Z. Dragisic, V. Ivanova, Get my pizza right: Repairing missing is-a relations in alc ontologies, in: H. Takeda, Y. Qu, R. Mizoguchi, Y. Kitamura (Eds.), Semantic Technology, Second Joint International Conference, JIST 2012, Nara, Japan, December 2-4, 2012. Proceedings, 2012, pp. 17–32. doi:10.1007/978-3-642-37996-3_2.
  • [46] J. Du, H. Wan, H. Ma, Practical tbox abduction based on justification patterns, in: S. P. Singh, S. Markovitch (Eds.), Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA., 2017, pp. 1100–1106.
  • [47] C. Pesquita, D. Faria, E. Santos, F. M. Couto, To repair or not to repair: reconciling correctness and coherence in ontology reference alignments, in: P. Shvaiko, J. Euzenat, K. Srinivas, M. Mao, E. Jiménez-Ruiz (Eds.), Proceedings of the 8th International Workshop on Ontology Matching, Vol. 1111 of CEUR Workshop Proceedings, 2013, pp. 13–24.
  • [48] J. Euzenat, P. Shvaiko, Ontology Matching, Springer, 2007.
  • [49] Y. Kalfoglou, M. Schorlemmer, Ontology mapping: the state of the art, The Knowledge Engineering Review 18(1) (2003) 1–31. doi:10.1017/S0269888903000651.
  • [50] N. F. Noy, Semantic integration: A survey of ontology-based approaches, SIGMOD Record 33(4) (2004) 65–70. doi:10.1145/1041410.1041421.
  • [51] P. Shvaiko, J. Euzenat, A survey of schema-based matching approaches, Journal on Data Semantics IV (2005) 146–171. doi:10.1007/11603412_5.
  • [52] P. Lambrix, L. Strömbäck, H. Tan, Information integration in bioinformatics with ontologies and standards, in: F. Bry, J. Maluszynski (Eds.), Semantic Techniques for the Web: The REWERSE perspective, Vol. 5500 of Lecture Notes in Computer Science, Springer, 2009, Ch. 8, pp. 343–376. doi:10.1007/978-3-642-04581-3_8.
  • [53] P. Shvaiko, J. Euzenat, Ontology matching: state of the art and future challenges, IEEE Transactions on Knowledge and Data Engineering 25(1) (2013) 158–176. doi:10.1109/TKDE.2011.253.
  • [54] Z. Dragisic, V. Ivanova, H. Li, P. Lambrix, Experiences from the anatomy track in the ontology alignment evaluation initiative, Journal of Biomedical Semantics 8 (1) (2017) 56:1–56:28. doi:10.1186/s13326-017-0166-5.
  • [55] P. Lambrix, R. Kaliyaperumal, A session-based ontology alignment approach enabling user involvement, Semantic Web Journal 8 (2) (2017) 225–251. doi:10.3233/SW-160243.
  • [56] C. Meilicke, H. Stuckenschmidt, A. Tamilin, Repairing ontology mappings, in: R. C. Holte, A. Howe (Eds.), Proceedings of the Twenty-Second AAAI Conference on Artificial Intelligence, July 22-26, 2007, Vancouver, British Columbia, Canada, 2007, pp. 1408–1413.
  • [57] Q. Ji, P. Haase, G. Qi, P. Hitzler, S. Stadtmüller, Radon - repair and diagnosis in ontology networks, in: The Semantic Web: Research and Applications. ESWC 2009., 2009, pp. 863–867. doi:10.1007/978-3-642-02121-3_71.
  • [58] C. Meilicke, Alignment incoherence in ontology matching, Ph.D. thesis, University of Mannheim (2011).
  • [59] E. Jiménez-Ruiz, B. C. Grau, Logmap: Logic-based and scalable ontology matching, in: L. Aroyo, C. Welty, H. Alani, J. Taylor, A. Bernstein, L. Kagal, N. F. Noy, E. Blomqvist (Eds.), The Semantic Web - ISWC 2011 - 10th International Semantic Web Conference, Bonn, Germany, October 23-27, 2011, Proceedings, Part I, 2011, pp. 273–288. doi:10.1007/978-3-642-25073-6_18.
  • [60] E. Jiménez-Ruiz, B. C. Grau, Y. Zhou, I. Horrocks, Large-scale interactive ontology matching: Algorithms and implementation, in: L. D. Raedt, C. Bessière, D. Dubois, P. Doherty, P. Frasconi, F. Heintz, P. J. F. Lucas (Eds.), ECAI 2012 - 20th European Conference on Artificial Intelligence. Including Prestigious Applications of Artificial Intelligence (PAIS-2012) System Demonstrations Track, Montpellier, France, August 27-31 , 2012, 2012, pp. 444–449. doi:10.3233/978-1-61499-098-7-444.
  • [61] E. Santos, D. Faria, C. Pesquita, F. M. Couto, Ontology alignment repair through modularization and confidence-based heuristics, PLoS ONE 10(12):e0144807) (2015) 1–19. doi:10.1371/journal.pone.0144807.
  • [62] A. Solimando, E. Jiménez-Ruiz, G. Guerrini, Detecting and correcting conservativity principle violations in ontology-to-ontology mappings, in: P. Mika, T. Tudorache, A. Bernstein, C. Welty, C. A. Knoblock, D. Vrandecic, P. T. Groth, N. F. Noy, K. Janowicz, C. A. Goble (Eds.), The Semantic Web - ISWC 2014 - 13th International Semantic Web Conference, Riva del Garda, Italy, October 19-23, 2014. Proceedings, Part II, 2014, pp. 1–16. doi:10.1007/978-3-319-11915-1_1.
  • [63] E. Jiménez-Ruiz, C. Meilicke, B. Cuenca Grau, I. Horrocks, Evaluating mapping repair systems with large biomedical ontologies, in: T. Eiter, B. Glimm, Y. Kazakov, M. Krötzsch (Eds.), Informal Proceedings of the 26th International Workshop on Description Logics, Ulm, Germany, July 23 - 26, 2013, 2013, pp. 246–257.
  • [64] C. Meilicke, H. Stuckenschmidt, A. Tamilin, Reasoning support for mapping revision, Journal of Logic and Computation 19 (5) (2009) 807–829. doi:10.1093/logcom/exn047.
  • [65] G. Qi, Q. Ji, P. Haase, A conflict-based operator for mapping revision, in: A. Bernstein, D. R. Karger, T. Heath, L. Feigenbaum, D. Maynard, E. Motta, K. Thirunarayan (Eds.), The Semantic Web - ISWC 2009, 8th International Semantic Web Conference, ISWC 2009, Chantilly, VA, USA, October 25-29, 2009. Proceedings, 2009, pp. 521–536. doi:10.1007/978-3-642-04930-9_33.
  • [66] J. Euzenat, Revision in networks of ontologies, Artificial Intelligence 228 (2015) 195–216. doi:10.1016/j.artint.2015.07.007.
  • [67] J. Du, G. Qi, X. Fu, A practical fine-grained approach to resolving incoherent OWL 2 DL terminologies, in: J. Li, X. S. Wang, M. N. Garofalakis, I. Soboroff, T. Suel, M. Wang (Eds.), Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, CIKM 2014, Shanghai, China, November 3-7, 2014, 2014, pp. 919–928. doi:10.1145/2661829.2662046.
  • [68] F. Baader, F. Kriegel, A. Nuradiansyah, R. Peñaloza, Making repairs in description logics more gentle, in: M. Thielscher, F. Toni, F. Wolter (Eds.), Principles of Knowledge Representation and Reasoning: Proceedings of the Sixteenth International Conference, KR 2018, Tempe, Arizona, 30 October - 2 November 2018., 2018, pp. 319–328.
  • [69] N. Troquard, R. Confalonieri, P. Galliani, R. Peñaloza, D. Porello, O. Kutz, Repairing ontologies via axiom weakening, in: S. A. McIlraith, K. Q. Weinberger (Eds.), Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, 2018, pp. 1981–1988.
  • [70] F. Baader, B. Ganter, B. Sertkaya, U. Sattler, Completing description logic knowledge bases using formal concept analysis, in: M. M. Veloso (Ed.), IJCAI 2007, Proceedings of the 20th International Joint Conference on Artificial Intelligence, Hyderabad, India, January 6-12, 2007, 2007, pp. 230–235.
  • [71] T. Dong, C. L. Duc, M. Lamolle, Tableau-based revision for expressive description logics with individuals, Jornal of Web Semantics 45 (2017) 63–79. doi:10.1016/j.websem.2017.09.001.
  • [72] J. Euzenat, Interaction-based ontology alignment repair with expansion and relaxation, in: C. Sierra (Ed.), Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, 2017, pp. 185–191. doi:10.24963/ijcai.2017/27.