Test-Driven Development of ontologies (extended version)

Emerging ontology authoring methods to add knowledge to an ontology focus on ameliorating the validation bottleneck. The verification of the newly added axiom is still one of trying and seeing what the reasoner says, because a systematic testbed for ontology authoring is missing. We sought to address this by introducing the approach of test-driven development for ontology authoring. We specify 36 generic tests, as TBox queries and TBox axioms tested through individuals, and structure their inner workings in an `open box'-way, which cover the OWL 2 DL language features. This is implemented as a Protege plugin so that one can perform a TDD test as a black box test. We evaluated the two test approaches on their performance. The TBox queries were faster, and that effect is more pronounced the larger the ontology is. We provide a general sequence of a TDD process for ontology engineering as a foundation for a TDD methodology.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

12/14/2018

More Effective Ontology Authoring with Test-Driven Development

Ontology authoring is a complex process, where commonly the automated re...
06/18/2019

A Framework for Parallelizing OWL Classification in Description Logic Reasoners

In this paper we report on a black-box approach to parallelize existing ...
11/23/2021

Link Analysis meets Ontologies: Are Embeddings the Answer?

The increasing amounts of semantic resources offer valuable storage of h...
06/20/2019

Generic Ontology Design Patterns at Work

Generic Ontology Design Patterns, GODPs, are defined in Generic DOL, an ...
08/06/2011

'Just Enough' Ontology Engineering

This paper introduces 'just enough' principles and 'systems engineering'...
11/23/2018

Competency Questions and SPARQL-OWL Queries Dataset and Analysis

Competency Questions (CQs) are natural language questions outlining and ...
05/30/2017

Preliminary results on Ontology-based Open Data Publishing

Despite the current interest in Open Data publishing, a formal and compr...

Code Repositories

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

The process of ontology development has progressed much over the past 20 years, aided by development information systems-oriented methodologies, such as methontology for a single-person/group monolithic ontology to ontology networks with NeOn [31], and both stand-alone and collaborative tools, such as Protégé [10] and MoKI [11]. They are examples of generic, high-level information systems-oriented methodologies, but support for effective ontology authoring—adding the right axioms and adding the axioms right—has received some attention only more recently. Processes at this ‘micro’ level of the development process, rather than the ‘macro’ level, may use the reasoner to propose axioms with FORZA [16], use Ontology Design Patterns (ODPs) [7], and start gleaning ideas from software engineering practices, notably exploring the notion of unit tests [35], eXtreme Design with ODPs [3], and Competency Question (CQ)-based authoring using SPARQL [28]

. However, testing whether a CQ can be answered does not say how to add/change the knowledge represented in the ontology, FORZA considers simple object properties only, and eXtreme Design limits one to ODPs that have to come from some place. Put differently, there is no systematic testbed for ontology engineering, to implement the CQ in the authoring process in a piecemeal fashion, other than manual efforts by a knowledge engineer to add or change something and running the reasoner to check its effects. This still puts a high dependency on expert knowledge engineering, which ideally should not be in the realm of an art, but rather at least a systematic process for good practices.

We aim to address this problem by borrowing another idea from software engineering: test-driven development (TDD). TDD ensures that what is added to the program core (here: ontology) does indeed have the intended effect specified upfront. Moreover, TDD in principle is cognitively a step up from the ‘just add stuff and lets see what happens’-attitude, therewith deepening the understanding of the ontology authoring process and the logical consequences of an axiom. In addition, it would make roll-backs and conflicting CQs easier to manage. At an informal, high level, one can specify the following three scenarios of usage.

  1. Developers (domain experts, knowledge engineers etc) specify CQs. A CQ is translated automatically into one or more axioms. This (these) axiom(s) are the input of the relevant TDD test(s) to be carried out. The developers who specify the CQs could be oblivious to the inner workings of the two-step process of translating the CQ and testing the axiom(s).

  2. The knowledge engineer knows which axiom s/he wants to add, types it, which is then fed directly into the TDD system.

  3. As there is practically only a limited amount of ‘types’ of axioms to add, one could create templates, alike the notion of the “logical macro” ODP [27]. For instance, a domain expert could choose the all-some template from a list, which then in the TDD system amounts to an axiom of the form . The domain expert instantiates it with relevant domain entities (e.g., ), and the related TDD test is then run automatically. The domain expert need not necessarily know the logic, but behind the usability interface, what gets sent to the TDD system is that axiom.

While in each case the actual testing can be hidden from the user’s view, it is necessary to specify what actually happens during such testing and how it is tested (in a similar way that it needed to be clear how the OWL reasoner works). Here, we assume that either the first step of the CQ process is completed, or the knowledge engineer adds the axiom, or that the template is populated, respectively; i.e., that we are at the stage where the axioms are fed into the TDD test system. To realise the testing, a number of questions have to be answered:

  1. Given the TDD procedure in software engineering—check that desired feature is absent, code it, test again—then what does that mean for ontology testing when transferred to ontology development?

  2. TDD requires so-called mock objects for ‘incomplete’ parts of the code, and mainly for methods; is there a parallel to it in ontology development, or can that aspect of TDD be ignored?

  3. In what way, and where, (if at all) can this be integrated as a methodological step in existing ontology engineering methodologies that are typically based on waterfall, iterative, or lifecycle principles rather than agile methodologies?

To work this out for ontologies, we take some inspiration from TDD for conceptual modelling. Tort et al. [34]

essentially specify ‘unit tests’ for each feature/possible addition to a conceptual model, and test such an addition against sample individuals. Translating this to OWL ontologies, such testing is possible by means of ABox individuals, and then instead if using an ad hoc algorithm, one can avail of the automated reasoner. In addition, for ontologies, one can avail of a query language for the TBox, namely, SPARQL-OWL

[18], and most of the test can be specified in that language as well. We define TBox and ABox-driven TDD tests for the basic axioms one can add to an OWL 2 DL ontology. Mock objects are required especially for TDD tests for the ‘RBox’, i.e., for the object property-specific axioms. To examine practical feasibility for the ontology engineer and determine which TDD strategy is the best option for ontology engineering, we first implemented this by means of a Protégé plugin as basic interface, and, second, evaluated the plugin on performance by comparing TBox and ABox TDD tests. TBox queries generally outperform the ABox ones, and this difference is more pronounced with larger ontologies. Finally, we outline a TDD ontology development methodology, which, while having overlap with some extant methodologies, cannot not neatly be squeezed into them. Overall, we thus add a new mechanism and tool to the ontology engineer’s ‘toolbox’ to enable systematic development of ontologies in an agile way.

The remainder of the paper is structured as follows. Section 2 describes related works on TDD in software and ontology development. Section 3 summarises the TDD tests and Section 4 evaluates them on performance with the Protégé plugin. We discuss in Section 5, where we also answer the above-mentioned research questions, and conclude in Section 6.

2 Related work

Before assessing related works in ontology engineering, we first describe some pertinent aspects of TDD from software engineering.

2.0.1 TDD in software development

The principal introduction of TDD is described in [2]: it is essentially a methodology of software development, where one writes new code only if an automated test has failed, and eliminating duplication in doing so. The test come from some place: a feature requirement should have a test specification to go with it, or, more concretely,“If you can’t write test for what you are about to code, then you shouldn’t even be thinking about coding.” [20]. That is, TDD permeates the whole of what was traditionally the waterfall or iterative methodology. Shrivastava and Jain [29] summed up the sequence as follows: 1) Write a test for a piece of functionality (that was based on a requirement), 2) Run all tests to check that the new test fails, 3) Write relevant code that passes the test, 4) Run the specific test to verify it passes, 5) Refactor the code, and 6) Run all tests to verify that the changes to the code did not change the external behaviour of the software (regression testing). The important difference with unit tests, is that TDD is a test-first approach rather than the more commonly known ‘test-last’ approach (design, code, test), and therewith moves into the realm of a methodology of its own permeating all steps in the development rather than only the testing phase with its unit tests for individual classes or components.

TDD is said to result in being more focussed, improved communication, improved understanding of required software behaviour, and reduced design complexity [20]. Quantitatively, TDD produced code passes more externally defined tests—i.e, better software quality—and involves less time spent on debugging, and experiments with students showed that the test-first group wrote more tests and were significantly more productive than the test-last group [14].

While all this focuses on the actual programming, the underlying ideas have been applied to conceptual data modelling [33, 34], resulting in a TDD authoring process, compared to, among others, Halpin’s conceptual schema design procedure [13]. Tort and Olivé reworked the test specification into a test-driven way of modelling, where each UML Class Diagram language feature has its own test specification in OCL that involves creating the objects that should, or ought not to, instantiate the UML classes and associations [34]. Also here, the test is supposed to fail first, then the model is updated, and then the test ought to pass. The tool that implements it was evaluated with modellers, which made clear, among others, that more time was spent on developing and revising the conceptual model, in the sense of fixing errors, than on writing the test cases [34].

2.0.2 Tests in ontology engineering

An early explorative work on borrowing the notion of testing from software engineering to apply it to ontology engineering is described in [35], which explores several adaptation options: testing with the axiom and its negation, formalising CQs, checks by means on integrity constraints, autoepistemic operators, and domain and range assertions. Working with CQs has shown to be, relatively, the most popular approach in the years since. A substantial step in the direction of test-driven ontology engineering was proposed by Ren et al [28], who analyse CQs for use with SPARQL queries that then would be tested against the ontology. It focuses on finding patterns in the natural language-based CQs, the sample SPARQL queries are querying for individuals only, and the formalisation stops at what has to be tested, not how that can, or should, be done. Earlier work on CQs and queries include the OntologyTest tool, which allows the user to specify tests to check the functional requirements of the ontology based on the CQs, using ABox instances and SPARQL queries [8]. Unlike extensive CQ and query patters, it specifies different types of tests focussed on the ABox rather than knowledge represented in the TBox, such as “instantiation tests” (instance checking) and “recovering tests” (query for a class’ individuals) and using mock individuals where applicable [8]; other instance-oriented test approaches have been proposed as well, although focussing on RDF/Linked Data rather than the OWL ABox [19]. A NeON plugin with similar functionality and methodical steps within the eXtreme Design approach also has been proposed [3], but not the types of tests. A more basic variant is incorporated in the EFO Validator111http://www.ebi.ac.uk/fgpt/sw/efovalidator/index.html for the Experimental Factor Ontology (EFO), which has tests for presence of a specific class or relation. Neither are based on the principle of TDD where, according to its specification, a test first has to fail, then the code is modified, and then the TDD test should pass. Warrender and Lord’s approach [36] does take that in consideration. They focus on unit tests for TBox testing where each query to the ontology requires a new test declaration. The tests have to be specified in Clojure with its own unfamiliar Tawny-Owl notation, describes only subsumption tests although the Karyotype ontology it is applied to is in , and the tests themselves are very tailored to the actual ontology rather than having reusable ‘templates’ for the tests covering all OWL language features. On the positive side, it can avail of some existing infrastructure for software testing rather than reinventing that technological wheel.

The ontology unit test notion of axiom and its negation of [35] has been used, in a limited sense, in advocatus diaboli, where in the absence of a disjointness axiom between classes, it shows the consequences to the modeller to approve or disapprove, and if the latter is selected, the tool adds the disjointness axiom [6]. Some of the OOPS! pitfalls that the software checks has some test-flavour to it [25], such as suggesting possible symmetry. FORZA shows the permissible relations one can add that will not lead to an inconsistency, which is based on domain and range constraints of properties and the selected entities one wants to relate [16]; that is, also not hypothesising a failure/absence of required knowledge in the TBox as such, though it can be seen as a variant of the domain & range assertions unit tests in [35].

Concerning overarching methodologies, none of the 9 methodologies reviewed by [9] are TDD-based, nor is the MeltingPoint García et al. propose themselves. A recent Agile-inspired tool and methodology is OntoMaven. Aspect OntoMaven [24] is an extension to OntoMaven that is based on reusing ideas from Apache Maven, advocated as being both a tool and supporting agile ontology engineering, such as COLM222http://www.corporate-semantic-web.de/colm.html. Regarding tests, besides the OntoMvnTest with ‘test cases’ for the usual syntax checking, consistency, and entailment, the documentation states it should be possible to reuse Maven plug-ins for further test types [24], but this has not been followed through yet. A different Agile-inspired method is, eXtreme Design with content ODPs [26], although this is also more a design method for rapid turnaround times rather than test-driven. Likewise, the earlier proposed RapidOWL is based on “iterative refinement, annotation and structuring of a knowledge base” [1] rather permeating the test-driven approach throughout the methodology. RapidOWL does mention the notion of “short releases”, which is very compatible with TDD cf. NeON’s waterfall-inspired ‘write many CQs first’ [31], but not how this is to be achieved other than publishing new versions quickly.

Thus, full TDD ontology engineering as such has not been proposed yet, to the best of our knowledge. While the idea of unit tests—which potentially could become part of TDD test—has been proposed, there is a dearth of actual specifications as to what exactly is, or should be, going on in such as test. It is also unclear whether even when one were to specify basic tests for each language feature, whether they can be put together in a modular fashion for the more complex axioms that can be declared with OWL 2. Further, there is no regression testing to check that perhaps an earlier modelled CQ—and thus a passed test—conflicts with a later one, and identifying which ones are conflicting.

3 TDD specification for ontologies

Before introducing the TBox and RBox TDD tests, first the general procedure in introduced and some clarifications are given on notation, the notion of true/false of a TDD test, and mock entities.

3.1 Preliminaries

Taking the TDD approach of devising a test that demonstrates absence of the feature (i.e., test failure), add feature, test again whether the test passes, the generalised TDD principle for ontologies then becomes:

  1. input: CQ and transform this into an axiom (optionally)

  2. given: axiom of type to be added to the ontology.

  3. check the vocabulary elements of are in ontology (itself a TDD test)

  4. run TDD test twice:

    1. the first execution should fail (check or not present),

    2. update the ontology (add ), and

    3. run the test again which then should pass (check that ) and such that there is no new inconsistency or undesirable deduction

  5. Run all previous successful tests, which still have to pass (i.e., regression testing)

In the following two subsections, we define such TDD test for TBox and RBox axioms with respect to OWL 2 DL features. For the TDD tests, there are principally two options: a TDD test at the TBox-level (where possible), or always using individuals explicitly asserted in the ABox. We specify tests for both approaches. For the test specifications, we use the OWL 2 standard notation for the ontology’s vocabulary: , , and .

For the TBox tests, we use SPARQL-OWL [18] where possible/applicable333While SPARQL-DL [30] might be more well-known than SPARQL-OWL, that version does not permit TBox queries with object properties, whereas SPARQL-OWL does, and we need those features. Its notation is principally reusing OWL functional syntax-style notation merged with SPARQL’s queried objects (i.e., ?x) for the formulation of the query, and adding a variable for the query answer; e.g., SubClassOf (?x D) will return all subclasses of class D. The query rewriting of SPARQL-OWL has been described in [18] and a tool implementing the algorithms, OWL-BGP, is freely available444https://github.com/iliannakollia/owl-bgp. While for some tests, one can use Protégé’s intuitive DL query tab to perform the tests, this is possible only for a fraction of the tests, and therefore omitted.

Some TBox and all ABox tests will require additional classes or individuals that only serve the purpose of testing and have to be removed after the test terminates successfully. This resembles the notion of mock objects in software engineering [21, 17], which we shall transport into the ontology setting. While TDD tests for ontologies do not need stubs for class methods, in some cases a test does need additional elements for it to be testable. Therefore, we do use mock class for a temporary OWL class, mock individual for an ABox individual created for the TDD test, and mock axiom for any auxiliary axiom that has to be added solely for testing purposes; they are to be removed from the ontology after completion of the test. (Possibly this can be taken further, using mock ontology modules, alike the mock databases [32], but this is not needed at this stage.)

Steps 3 and 4a in the sequence listed above may give an impression of epistemic queries. It has to be emphasised that there is a fine distinction between 1) checking when an element is in the vocabulary of the TBox of the ontology (in or ) versus autoepistemic queries, and 2) whether something is logically true or false versus a test evaluating to true or false. Proposals for epistemic extensions for OWL exist, which concern instance retrieval and integrity constraints checking [22] that reduces the K-operator to plain OWL queries, or closed world reasoning [12]. The TDD test focus in step 3, above, is of a distinctly different flavour. We need to know whether an element is present in the ontology, but we do not want to ‘know’ or assert things about individuals (say, that when there is an instance of country, that there must also be an object of capital associated with it, as in, KCountry AhasCapital.ACapital). As such, an epistemic query language is not needed for the TBox-level axioms. In the TDD context, the epistemic-sounding ‘not asserted in or inferred from the ontology’ is to be understood in the context of a TDD test, like whether an ontology has some class in its vocabulary, not whether it is ‘known to exist’ in one’s open or closed world based on the knowledge represented in the ontology.

3.2 Test patterns for TBox axioms

The tests are introduced in sequence, where the primed test names are the ones with explicit individuals.

3.2.1 Class subsumption, or .

When the axiom to add is of type , then should be true if it were not present. Logically, this means should be inconsistent, i.e., . Given the current Semantic Web technologies, it is easier to just query the ontology for the superclasses of and to ascertain that is not in query answer rather than create and/or execute tailor-made tableau algorithms. Thus, a test can be true or false. In SPARQL-OWL notation and quasi algorithm for comprehensiveness, is:

1:Test i.e.,
2: SubClassOf(?x D)
3:if  then is neither asserted nor entailed in the ontology
4:     return is false
5:else
6:     return is true
7:end if

After adding to the ontology, the same test is run, which should evaluate to and therewith is true.

The TTD test with individuals concerns a check whether an instance of is also an instance of . That is, for we have:

1:Test i.e.,
2:Create a mock object
3:Assert
4: Type(?x D)
5:if  then is neither asserted nor entailed in the ontology
6:     return is false
7:else
8:     return is true
9:end if

3.2.2 Class disjointness, or .

One can assert the complement, , or disjointness, . Let us consider the former first, for a test . For the test, then should be true, or false (in the sense of ‘not be in the ontology’), Testing for the latter only does not suffice, however, as there are more cases where holds, but disjointness is not really applicable—being classes in distinct sub-trees in the TBox—or holds when disjointness is asserted already, which is when and are sibling classes. For this complement with the inclusion axiom in OWL, we simply can query for the complement in the ontology:

1:Test i.e., test
2: ObjectComplementOf(C ?x)
3:if  then thus
4:     return is false
5:else
6:     return is true
7:end if

Concerning the stronger version of disjointness, , in SPARQL-OWL notation:

1:Test i.e., test
2: DisjointClasses(?x D)
3:if  then thus,
4:     return is false
5:else
6:     return is true
7:end if

The second option for the test, , is to involve the ABox and use a query or run the reasoner. The sequence of steps is as follows, availing of the reasoner only and no additional queries are necessary:

1:Test or i.e., test
2:Create individual that is, is a mock object
3:Assert and ;
4: Run the reasoner
5:if  == consistent then the test fails, i.e., then either or directly or through one or both of their superclasses
6:     return or is false
7:else the ontology is inconsistent, the test passed; thus either or is already asserted among both their superclasses or among or and a superclass of or , respectively.
8:     return or is true
9:end if

Further, from a modelling viewpoint, it would make sense to also require and to be siblings. The sibling requirement can be added as an extra check in the interface to alert the modeller to it, but not be enforced from a logic viewpoint.

3.2.3 Class equivalence, and .

When the axiom to add is of the form , then should be true before the edit, or false. The latter is easier to test, for we can simply run aforementioned test twice, once for and once for : if both are true, then , if one of them or neither is true, then . More succinctly though, one can use the following SPARQL-OWL query:

1:Test i.e., test
2: EquivalentClasses(?x D)
3:if  then thus,
4:     return is false
5:else
6:     return is true
7:end if

Subsequently, it has to be added to the ontology, queried again, and evaluate to . However, this works only when is an atomic class, not when is a complex one.

For with individuals, we specify an extended version of as follows:

1:Test i.e., test
2:Create a mock object
3:Assert
4: Type(?x D)
5:if  then thus,
6:     Delete and
7:     return is false
8:else
9:     Delete
10:     Assert
11:      Type(?x C)
12:     if  then thus,
13:         return is false
14:     else
15:         return is true
16:     end if
17:     Delete and
18:end if

3.2.4 Simple existential quantification, or .

Let the axiom type be of the form , then should be true, or false (or: not asserted) before the ontology edit. One could do a first check that is not a descendant of but if it is, then it may be the case that , with a different class from . This still requires one to confirm that is not a subclass of . In SPARQL-OWL, we can combine this into one query/TDD test:

1:Test i.e., test
2: SubClassOf(?x ObjectSomeValuesFrom(R D))
3:if  then thus,
4:     return is false
5:else
6:     return is true
7:end if

If the test passes, i.e., , then the axiom is to be added to the ontology, the query run again, and if , then the test cycle is completed.

From a cognitive, or modelling, viewpoint, desiring to add a CQ that amounts to (‘each has an outgoing arc to anything that is not a ’) may look different, but , so it amounts to testing , i.e., essentially the same pattern. This also can be formulated directly into a SPARQL-OWL query, encapsulated in a TDD test:

1:Test i.e., test
2: SubClassOf(?x ObjectSomeValuesFrom(R ObjectComplementOf(D)))
3:if  then thus,
4:     return is false
5:else
6:     return is true
7:end if

It is slightly different for (‘all s do not have an outgoing arc to ’). The query with TDD test is as follows:

1:Test i.e., test
2: SubClassOf(?x ObjectComplementOf(ObjectSomeValuesFrom(R D)))
3:if  then thus,
4:     return is false
5:else
6:     return is true
7:end if

hence, then the test fails, so it can be added, the test runs gain, and then passes. The TDD test with individuals can be carried out as follows.

1:Test i.e., test
2:Create two mock objects, and
3:Assert , , and
4: Type(?x, ObjectSomeValuesFrom(R D))
5:if  then thus,
6:     return T() is false
7:else
8:     return T() is true
9:end if
10:Delete , , , , and

The two negated cases are as follows.

1:Test i.e., test
2:Create two mock objects, and
3:Assert , , and
4: Type(?x, ObjectSomeValuesFrom(R ObjectComplementOf(D)))
5:if  then thus,
6:     return T() is false
7:else
8:     return T() is true
9:end if
10:Delete , , , , and
1:Test i.e., test
2:Create two mock objects, and
3:Assert , , and
4: Run the reasoner
5:if  consistent then thus,
6:     return T() is false
7:else
8:     return T() is true
9:end if
10:Delete , , , , and

3.2.5 Simple universal quantification, or .

Let be , then should hold, or false (not be present in the ontology), before the ontology edit. This has a similar pattern to the one for existential quantification,

1:Test i.e., test
2: SubClassOf(?x ObjectAllValuesFrom(R D))
3:if  then thus,
4:     return is false
5:else
6:     return is true
7:end if

which then can be added and the test ran again. The TDD test for with individuals is much alike :

1:Test i.e., test
2:Create two mock objects, and  
3:Assert , , and
4: Type(?x, ObjectAllValuesFrom(R D))
5:if  then thus,
6:     return T() is false
7:else
8:     return T() is true
9:end if
10:Delete , , , , and

3.3 Test patterns for object propery axioms

From here onward, the tests deal with object properties more specifically—sometimes called the RBox—that more often than not do not lend themselves well for easy querying in the DL query tab, for the DL query tab returns results on classes and individuals only, because DL query is essentially a class expression language, i.e., it can only express (complex) classes. However, SPARQL-OWL and the automated reasoner can be used fairly straightforwardly, so it is essentially an interface limitation that can be solved with adding a new plugin for TDD.

3.3.1 Domain axiom, or .

Let be that is not yet in . To verify that it is not, should be true, or false. With SPARQL-OWL, there are two options. First, one can query for the domain:

1:Test i.e., test
2: ObjectPropertyDomain(R ?x)
3:if  then thus,
4:     return is false
5:else
6:     return is true
7:end if

Alternatively, one can query for the superclasses of (noting that it is shorthand for ), where the above-listed query in the TDD is replaced with:
SubClassOf(SomeValuesFrom(R Thing) ?x)
Note that only will be returned if is the only domain class of or when (but not if it is , which is a superclass of ). Alternatively, with the individuals:

1:Test i.e., test
2:Check and
3:Add individuals and
4:Add as object property assertion
5:Run the reasoner
6:if  then (also in the strict sense as is or with a conjunction), hence then the test fails as intended.
7:     return is false
8:else
9:     return is true
10:end if
11:Delete individuals and

If the answer is empty, then does not have any domain specified yet, and if , then , so that it can be added and the test run again to complete the cycle.

3.3.2 Range axiom, or .

When is a range axiom, should not be in the ontology before the TDD test. This is similar to the domain axiom test:

1:Test i.e., test
2: ObjectPropertyRange(R ?x)
3:if  then thus,
4:     return is false
5:else
6:     return is true
7:end if

or, in the second option, to replace the TDD query with
SubClassOf(SomeValuesFrom(ObjectInverseOf(R) Thing) ?x)
The answer will be if or , and only owl:Thing if no range was declared for . The test with individuals is as follows:

1:Test i.e., test
2:Check and
3:Add individual and to the ABox
4:Add as object property assertion
5:if  then , as intended
6:     return is false
7:else
8:     return is true
9:end if
10:Delete and individuals and

3.3.3 Object property subsumption and equivalence, and , and and .

When axiom type is a property subsumption, , then we have to test that , or that fails. The SPARQL-OWL query in the is as follows:

1:Test i.e., test
2: SubObjectPropertyOf(?x S)
3:if  then thus,
4:     return is false
5:else
6:     return is true
7:end if

One cannot query this as such in Protégé’s DL query tab, other than by using individuals, i.e., a version of . With the OWL semantics, for to hold, it means that, given some individuals and , that if then . The desired result is computed by the reasoner anyhow. The steps for the TDD test:

1:Test i.e., test
2:Check
3:Add individuals to the ABox, add
4:Run the reasoner
5:if  then (practically: not shown in the “property assertions” in the individuals tab for , with “Show inferences” checked), thus
6:     return is false
7:else
8:     return is true
9:end if
10:Delete , and individuals and

Then the modeller would add , run the test again, and it should then infer , as . This, however, does not guarantee was added, and not inadvertently . Their difference can be easily observed with the following set-up:

1:Test i.e., test
2:Check
3:Add mock individuals to the ABox
4:Add and as object property assertion
5:Run the reasoner
6:if  and  then , hence the ontology edit executed correctly
7:     return is false
8:else i.e. , so
9:      is true
10:end if
11:Delete and , and

For object property equivalence at the Tbox level, i.e., , one could use twice, or simply use the EquivalentObjectProperties with SPARQL-OWL:

1:Test i.e., test
2: EquivalentObjectProperties(?x S)
3:if  then thus,
4:     return is false
5:else
6:     return is true
7:end if

3.3.4 Object property inverses, and .

There are two options here since OWL 2, and it is hard to choose which one is ‘better’: using explicit inverses tends to be chosen for understanding (e.g., teaches with inverse declared explicitly as taught by), whereas using an ‘implicit’ inverse (e.g., teaches and teaches) improved reasoner performance in at least one instance [15]. Practically, for the failure-test of TDD, we can test only the former case, as the latter is only used in axioms. Also here one can choose between a TBox or an ABox approach. The TBox approach with a SPARQL-OWL query for :

1:Test i.e., test
2: InverseObjectProperties(?x S)
3:if  then thus,
4:     return is false
5:else
6:     return is true
7:end if

Using the Abox, we again have to work with mock objects:

1:test i.e., test
2:Check
3: Assume is intended to be the inverse of (with and having different names), and we check for its absence:
4:Add mock individuals to the ABox
5:Add as object property assertion
6:Run the reasoner
7:if  then , hence the test fails, as intended
8:     return is false
9:else
10:     return is true
11:end if
12:Delete mock individuals and

Then add , run the test again, which then should evaluate to true.

3.3.5 Object property chain, or .

When is one of the permissible chains (except for transitivity; see below), such as , , (with ). This is increasingly more cumbersome to test, for the simple fact that many more entities are involved, hence, more opportunity to have incomplete knowledge represented in the ontology and, hence, more hassle to find all the permutations that lead to not having the desired effect. Perhaps the easiest way to check whether the ontology has the property chain, is to search the owl file for owl:propertyChainAxiom, with the relevant properties included in order, or, for that matter, the SPARQL-OWL query in the TDD test:

1:Test i.e., test
2: SubObjectPropertyOf(ObjectPropertyChain(R S) ?x)
3:if  then thus,
4:     return is false
5:else
6:     return is true
7:end if

and similarly with the other permutations. However, simply checking the owl file or executing the query misses three aspects of chains: i) there is no point in having a property chain if the properties involved are never used in the intended way anyway, 2) this cannot ascertain that it does only what was intended, and 3) whether the chain is allowed in OWL 2, i.e., not violating ‘interfering’ constraints (the properties have to be ‘simple’). For to be interesting for the ontology, also at least one and one should be present. If they all were, then a SPARQL-OWL query
SubClassOf(?x ObjectSomeValuesFrom(S E))
will have . If either of the three axioms are not present, then . From the perspective of the ABox testing, we need the following sequence of steps:

1:Test i.e., test
2:Check and
3:if  then
4:     Add the missing class(es) (, , and/or ) as mock classes
5:end if
6:Run the test or , for both and for
7:if  is false then
8:     Add , , or both, as mock axiom
9:end if
10:if  then then test is meaningless, for it would not test the property chain
11:     Add mock class , mock axiom
12:     Verify with or
13:      SubClassOf(?x ObjectSomeValuesFrom(S E))
14:     if  then thus,
15:         return is false
16:     else
17:         return is true
18:     end if
19:else so,
20:      SubClassOf(?x ObjectSomeValuesFrom(S E))
21:     if  then thus,
22:         return is false
23:     else
24:         return is true
25:     end if
26:end if
27:Delete all mock objects, classes, and axioms

Assuming that the test fails, i.e., (resp. ) and thus , then add the property chain and run the test again, which then should pass (i.e., ). When it does, any mock classes and axioms should be removed.

The procedure holds similarly for the other permissible combinations of object properties in a property chain/complex role inclusion.

3.3.6 Object property characteristics, .

Testing absence/presence of the object property characteristics is surely feasible with mock individuals in the ABox, but is doable only for transitivity and local reflexivity in the TBox.

is functional, , i.e., some object has at most one individual -successor. The TDD test procedure is as follows.

1:Test i.e., test
2:Check and ; if not present, add.
3:Assert mock axioms , , and , if not present already.
4:Run reasoner
5:if  is consistent then thus,
6:     return is false
7:else is inconsistent
8:     return is true
9:end if
10:Remove mock axioms and individuals, as applicable

is inverse functional, . This is as above, but then in the other direction, i.e., , with declared distinct, will result in a consistent ontology when . Thus:

1:Test i.e., test
2:Check and ; if not present, add.
3:Assert mock axioms , , and , if not present already.
4:Run reasoner
5:if  is consistent then thus,
6:     return is false
7:else is inconsistent
8:     return is true
9:end if
10:Remove mock axioms and individuals, as applicable

is transitive, or , so that with and , it will infer . As with the object property chain test (), this object property characteristic is only ‘interesting’ for the ontology engineer if there are at least two related axioms so that one obtains a non-empty deduction thanks to the transitive object property. Further, it is the only object property characteristic that has a real effect in the TBox. If the relevant axioms are not asserted, they have to be added.

1:Test i.e., test
2:Check and
3:if  then
4:     Add the missing class(es) (, , and/or as mock classes)
5:end if
6:if  and are not asserted then
7:     add and to
8:end if
9: SubClassOf(?x ObjectSomeValuesFrom(R E))
10:if  then thus,
11:     return is false
12:else
13:     return is true
14:end if
15:Remove mock classes and axioms, as applicable

The ABox-based test is as follows.

1:Test i.e., test
2:Check , . If not, introduce as mock objects.
3:Assert mock axioms and , if not present already.
4:Run reasoner
5:if  then thus,
6:     return is false
7:else
8:     return is true
9:end if
10:Remove mock individuals

is symmetric, , so that with , it will infer . The test-to-fail—assuming —is as follows.

1:Test i.e., test
2:Check . Introduce as mock objects ().
3:Assert mock axiom .
4: ObjectPropertyAssertion(R x? a)
5:if  then thus,
6:     return is false
7:else
8:     return is true
9:end if
10:Remove mock assertions and individuals

Alternative to the query in line 4, one can check in the ODE whether is inferred (yellow in the Protégé Individuals tab).

is asymmetric, . This is easier to test with the negative, i.e., assert objects symmetrically and distinct, and if the ontology is not inconsistent, then . More precisely:

1:Test i.e., test
2:Check . Introduce as mock objects ().
3:Assert mock axioms and .
4:Run reasoner
5:if  not inconsistent then thus,
6:     return is false
7:else
8:     return is true
9:end if
10:Remove mock axioms and individuals

is reflexive, or . The object property can be either globally reflexive (Ref()), or locally (). Global reflexivity is typically not what one wants, but if in the exceptional case the modeller does, then the following test should be executed.

1:Test Ref() i.e., test
2:Check .
3:Introduce as mock objects ().
4:Run the reasoner
5:if  then thus,
6:     return Ref() is false
7:else
8:     return Ref() is true
9:end if
10:Remove mock object

And adding Ref() will have the test evaluate to true. Local reflexivity uses Self in an axiom; that is, we need to check whether . This is essentially the same as but then with Self instead of the explicit class, i.e.:

1:Test i.e., test
2: SubClassOf(?x ObjectSomeValuesFrom(R Self))
3:if  then thus,
4:     return is false
5:else
6:     return is true
7:end if

or, in ABox variant:

1:Test i.e., test
2:Check . Introduce as mock objects ().
3:Assert mock axiom
4: Type(?x C), PropertyValue(a R ?x)
5:if  then thus,
6:     return is false
7:else
8:     return is true
9:end if
10:Remove and mock object

is irreflexive, . As with asymmetry, this is easier to test with the converse: assert , run the reasoner, then the ontology is consistent (i.e., then ). Add , run the reasoner, then , and finally remove mock individual and assertion.

1:Test i.e., test
2:Check , and add
3:Assert mock axiom
4:Run reasoner
5:if  is consistent then thus,
6:     return is false
7:else is inconsistent
8:     return is true
9:end if
10:Remove mock axiom and individual, as applicable

This concludes the basic tests. While the logic permits that some class on the left-hand side of the inclusion axiom may be a complex and defined concept, we do not consider such cases here, as due to the tool design of the most widely used ODE, Protégé, the left-hand side of the inclusion axiom has only a single class .

4 Experimental evaluation with a Protégé plugin for TDD

We describe the design of the plugin and evaluation of the performance tests in this section.

4.1 Design

In order to support ontology engineers in performing TDD, we have implemented the Protégé plugin named TDDOnto. The plugin provides a view where the user may specify the set of tests to be run. After their execution, the status of the tests is displayed. It is also possible to add a selected axiom to the ontology (and re-run the test). Fig. 1 presents the screenshot of the TDDOnto plugin.

Figure 1: A screenshot of the TDDOnto plugin.

4.2 Evaluation of the TBox and ABox based TDD tests

The aim of the evaluation is to assess which option is the more feasible one. This can be done from a plugin usability viewpoint and from the algorithmic viewpoint, in the sense of which type of test is faster (though slow response times slow down the ontology authoring process and is also a dimension of usability). We take the latter option here. The question, then, is Which TDD approach—queries or mock objects—is better? We describe the set-up of the experiment first, and then proceed to the results and discussion.

4.2.1 Set-up of the experiment.

To assess this quantitatively, we pose the following two general hypotheses:

  1. Query-based TDD is faster overall.

  2. Classification time of the ontology contributes the most to overall performance (time) of a TDD test.

The reason why we think that H1 will hold is because once classified, one can query multiple times without having to classify the ontology again, and for some mock-object TDD tests the ontology should be inconsistent, which is a more cumbersome step to deal with than checking membership of a class or individual in a query answer. The reason for expecting H2 to hold is that the other operations—adding and removing entities, testing for membership—can be executed in linear time, whereas there are not many ontologies in a language that is linear or less in data complexity. These general hypotheses can be refined to suit statistical tests for each hypotheses:

  1. There is no difference between query-based and mock-object based TDD tests.

  2. There is a difference, with query-based having lower values for time taken to execute the tests.

  1. TDD overall execution times are arbitrarily subdivided into ontology classification time and the TDD test part.

  2. There is a difference, with ontology classification time taking much more () of the TDD overall execution times than the TDD test part.

The performance is expected to depend on the ontology’s content that is being revised, as reasoning time does. It is unclear whether the overall TDD test execution time and what is attributed to plain ontology classification—also depends on the characteristics of the ontology. If it does, it is due to something internal to the reasoner, to which we do not have access. Notwithstanding, we would like to obtain an indication whether there might be interference regarding this aspect. Therefore we categorise test ontologies into groups depending on the number of their axioms.

Testing for/adding the axioms to the selected ontologies can be done in two ways: adding new elements, or reusing elements of the ontology. The former is certainly easier to carry out, the latter is truer to the intended use of a TDD test. In the experiments we followed the latter option and randomly selected existing ontology classes and properties for the tests.

4.2.2 Materials and methods.

The results presented in this section are computed based on the tests perfomed on 67 OWL ontologies from the TONES repository 555http://rpc295.cs.man.ac.uk:8080/repository/browser, downloaded from the mirror available at OntoHub 666https://ontohub.org/repositories. We selected those ontologies from all available TONES ontologies, while omitting the other ones that were either not in OWL (but in OBO format) or were having datatypes incompatible with OWL 2, causing exceptions of the reasoner or out of memory exceptions. The results were computed on Mac Book Air with 1.3 GHz Intel Core i5 CPU and 4 GB RAM. As an OWL reasoner, we used the same reasoner that is built-in into OWL-BGP, namely HermiT 1.3.8.

The tests were generated randomly, and each test kind was repeated 3 times to obtain more reliable results. We divided our ontologies into 4 groups, depending on the overall number of their axioms: up to 100 (20 ontologies), 100–1000 axioms (35 ontologies), 1000–10,000 axioms (10 ontologies), and over 10,000 (2 ontologies). All the experimental results are available at https://semantic.cs.put.poznan.pl/wiki/aristoteles/doku.php.

4.2.3 Results and discussion.

During our experiments we also found out that not all the features of OWL 2 are covered by OWL-BGP. In particular, RBox axioms (e.g., subPropertyOf) and property characteristics were not handled. Therefore, we only present the comparative results of the tests that could be run in both settings: ABox tests and TBox SPARQL-OWL tests.

The statistics are presented in the Fig 2, where X axis presents the groups of the ontologies (the ranges of the minimum and maximum number of the axioms each ontology in the group has). Note that the Y axis is scaled logarithmically. On the figure, there is a box plot presenting for every group of ontologies: the median

(horizontal line within the box); the first and third quartile (bottom and top line of the box); the lowest value above

(short horizontal line below the box), the highest value below (short horizontal line above the box), where

(interquartile range) is represented with the height of the box; and outliers (points above and below of the short lines).

From the results as displayed in Fig. 2, it follows that TBox (SPARQL-OWL) tests are generally faster than the ABox ones, and these differences are larger in the sets of larger ontologies. It is also apparent that the ontology classification times are large—in fact, higher on average—in comparison to the times of running the test.

In the Fig. 3, we present the running times per test type and the kind of the tested axiom. Again, the TBox based method is generally faster, with an exception of testing disjointness.

Figure 2: Test computation times per test type (ABox versus TBox based SPARQL-OWL) and ontology axiom count.
Figure 3: Test computation times per test type (ABox versus TBox based SPARQL-OWL) and per the kind of the tested axiom.

We have also tested two alternative TBox querying approaches (based on SPARQL-OWL and based on using OWL API and the reasoner). The results of this comparison between SPARQL-OWL and OWL API are presented in Figures 4 and 5 and showing even better performance of the TBox TDD tests.

Figure 4: Test computation times per test type (OWL API versus SPARQL-OWL TBox test) and ontology axiom count.
Figure 5: Test computation times per test type (OWL API versus SPARQL-OWL TBox test) and per the kind of the tested axiom.

5 Discussion

One might have wondered: why not simply browse the ontology to see if the axiom is already there? There are several reasons that the browsing-way-out is not ideal. First, then one does not know the implications (unless first classified). Second, browsing large ontologies is cumbersome, and more easily results in cognitive overload that hampers speedy ontology development. Third, the browsing is purely manual checking, making it easy to overlook something. The process of finding and adding an axiom is amenable to automation that solves these three issues. Instead of manual checks and sample axioms that have to be all manually added and possibly removed during the regular ‘add something and lets see what happens’, this can be managed in one fell swoop. In addition, the TDD tests give also the convenience and benefit of systematic regression testing. There are some hurdles to it realisation, however, which we discuss in the next subsection. Afterward, we outline how a TDD methodology for ontology engineering may look like in general terms.

5.1 Reflections on specifying and implementing a TDD tool

Aside from the consideration whether one would have to go the way of epistemic queries, the first main aspect was how to specify the TBox tests. For instance, some tests can be done easily with the DL query tab in Protégé; e.g. ’s SPARQL-OWL query amounts to either:
C select Super classes
or, in the other direction:
D select Sub classes
without even having to face the consideration of having blank nodes/unnamed classes (complex class expressions) on the right-hand-side of the inclusion axiom that was not supported in Kollia et al’s BGP [18]. The DL Query tab interface does not have some of the required functionality, notably regarding tests for object properties, it would have to be redesigned to function as TDD test interface anyway, and it is not ideal for TDD test performance testing due to unclear times taken up by the Protégé interface processing. Therefore, these DL Query tab specifications have been omitted from this paper.

The more principled aspect underlying the TDD test realisation, however, is the technique to obtain the answer of a TDD test: SPARQL SELECT-queries, SPARQL-OWL’s BGP that uses SPARQL answering engine and HermiT v1.3.8, SPARQL-DL with ASK queries also using the OWL API and one of the OWL 2 DL reasoners. While the difference in performance between the ABox test and TBox tests are generally explainable—the former always modifies the ontology, so requires an extra classification step—it is not for the outlier (disjointness) or why in some cases the difference is larger (subsumption, equivalence) than in others (queries with quantifiers). Further, it may be the case that overall performance may be different when a different reasoner is used, as reasoners do differ [23]. Likewise, we observed a trend towards bigger differences ABox vs SPARQL testing with larger ontologies. We do not aim to address this issue here, but note it for further investigation into the matter.

A related issue is the maturity of the tools used for the performance evaluation. Of the ontologies selected for testing, several returned errors, which were due to incompatible data types of an OWL DL ontology with OWL 2 DL-tailored tools. Further, especially querying in the context of the TDD tests for object properties faced limitations, as most of the required features were not implemented777it is, however, possible to carry out the sequence of each of the ABox test ‘manually’ by adding the individuals, relations, run the reasoner and check the instance classification results.. This forced us to redesign the experiment into one of ‘test what can be done’ now and infer tendencies from that so as to have a solid, experimentally motivated, basis for deciding which technique likely will have the best chance of success, hence, the best candidate for extending the corresponding tool. This appeared to be indeed preferring TBox tests over ABox tests, although most of the RBox test will have to be carried out as ABox tests.

Finally, once all test are implemented and a multi-modal interface developed to cater for the three principal use case scenarios and any other requirements emanating from the TDD methodology (see next section), user evaluations are to be conducted to evaluate whether also for ontology engineering the TDD benefits can be reaped, as observed for conceptual modelling and software development.

5.2 A step toward a TDD ontology engineering methodology

A methodology is a structured collection of, principally, methods and techniques, processes, people having roles possibly in teams, and quality measures and standards across the process (see, e.g., [4]), and has been shown to improve the overall results compared to doing something without a methodology. This is not to say that when a full-fledged TDD ontology engineering methodology has been developed, it should be the ontology engineering methodology. Like for software engineering’s ‘methodology selection grid’ [4], it will exceedingly suit some ontology development projects but perhaps not others. Here, with the previously described test specifications and their experimental evaluation only, we do not purport to have a full TDD methodology, but a foundational step in that direction that indicates where and how it differs in design compared to the typical waterfall, iterative, or lifecycle-based methodologies. We rework the software development TDD procedure (recall Section 2) into a sequence of steps applicable to ontology engineering, as follows:

  1. Choose the usage scenario as outlined in Section 1: CQ-driven TDD (formalised CQ specification translated into an axiom); Authoring-driven knowledge engineer (the axiom one wants to add), or Authoring-driven domain expert (a selected template populated);

  2. (Have) Select(ed) a test for the axiom;

  3. Run test to check that it fails;

  4. Write relevant knowledge in the ontology that should pass the test, i.e., add classes, object properties, or axioms, as applicable (this may simply be clicking a button to add the formalised CQ, the provided axiom, or the filled-in template);

  5. Classify the ontology to check that nothing contradictory was added;

  6. Run the same test as before, to verify it passes;

  7. Optionally refactor the formally represented knowledge (including classification and checking there are no undesirable deductions, and possibly asserting the implicit knowledge);

  8. Run all tests to verify that the changes to the ontology did not change the intended deductions from the ontology (regression testing); resolve any conflicting axioms or CQs (possibly by another TDD cycle).

A sketch of the process is depicted in Fig. 6. It is possible to refine these steps further, such as a way to manage the deductions following from having added new knowledge and how to handle an inconsistency or undesirable deduction due to contradictory CQs (alike resolving conflicting requirements in software engineering) that may surface in steps 5 and 8. These detailed aspects are left for future work.

Figure 6: Sketch of a possible ontology lifecycle that focuses on TDD, and the steps of the TDD procedure summarised in key terms.

5.3 Answering the research questions

The answers to the questions posed in Section 1 can be summarised as follows.

The first question was formulated rather broadly: Given the TDD procedure in software engineering, then what does that mean for ontology testing when transferred to ontology development? The main answer to this is the specification of tests for each type of axiom one can add to the ontology, which can be realised in different ways, namely, queries over the TBox and through individuals in the ABox. While the general idea is thus the same—requirement, test specification, test fails, change something, run test again to check it passes—the means of conducting a test for ontologies is thus different. One does not check code ‘functionality’ but whether the some piece of knowledge is present and represented in the way as intended.

Regarding the second question TDD requires so-called mock objects for ‘incomplete’ parts of the code, and mainly for methods; is there a parallel to it in ontology development, or can that aspect of TDD be ignored? can be answered in the affirmative. In particular for the general ABox tests and the so-called RBox, this ‘mock’ thing had to be refined into mock individuals, mock classes, and mock axioms. The experimental evaluation showed this approach is not as fast as TBox tests, but there is no avoiding some ABox test especially when testing most of the object property characteristics.

Last, In what way, and where, (if at all) can this be integrated as a methodological step in existing ontology engineering methodologies that are typically based on waterfall, iterative, or lifecycle principles rather than agile methodologies? TDD ontology engineering, like TDD for software development, has its own procedure. While some aspects overlap, such as CQs (requirements), and the ‘formalisation’ step in methontology [5] (writing the axioms, by the knowledge engineer), both can be avoided as well: the former by the engineering, the latter by the domain expert. With some stretching of the notion of ‘lifecycle’, the lifecycle is one of a TDD cycle only, which can be part of a larger lifecycle of multiple TDD cycles. There is no single neat ‘plug-in’ point for TDD into the waterfall and iterative methodologies, however.

6 Conclusions

This paper introduced 36 tests for Test-Driven Development of ontologies, specifying what has to be tested, and how. Tests were specified both at the TBox-level with queries and for ABox individuals, using mock objects. The implementation of the main tests demonstrated that the TBox test approach performs better.A high-level 8-step process for TDD ontology engineering was proposed. Future work pertains to extending tools to also implement the remaining tests, elaborate on the methodology, and conduct use-case evaluations.

6.0.1 Acknowledgments

This research has been supported by the National Science Centre, Poland, within the grant number 2014/13/D/ST6/02076.

References

  • [1] Auer, S.: The RapidOWL methodology–towards Agile knowledge engineering. In: Enabling Technologies: Infrastructure for Collaborative Enterprises, 2006. WETICE ’06. 15th IEEE International Workshops on. pp. 352–357 (June 2006)
  • [2] Beck, K.: Test-Driven Development: by example. Addison-Wesley, Boston, MA (2004)
  • [3] Blomqvist, E., Sepour, A., Presutti, V.: Ontology testing – methodology and tool. In: 18th International Conference on Knowledge Engineering and Knowledge Management (EKAW’12). LNAI, vol. 7603, pp. 216–226. Springer (2012)
  • [4] Cockburn, A.: Selecting a project’s methodology. IEEE Software 17(4), 64–71 (2000)
  • [5] Fernández, M., Gómez-Pérez, A., Pazos, A., Pazos, J.: Building a chemical ontology using METHONTOLOGY and the ontology design environment. IEEE Expert: Special Issue on Uses of Ontologies January/February, 37–46 (1999)
  • [6] Ferré, S., Rudolph, S.: Advocatus diaboli – exploratory enrichment of ontologies with negative constraints. In: ten Teije, A., et al. (eds.) 18th International Conference on Knowledge Engineering and Knowledge Management (EKAW’12). LNAI, vol. 7603, pp. 42–56. Springer (2012), oct 8-12, Galway, Ireland
  • [7] Gangemi, A., Presutti, V.: Ontology design patterns. In: Staab, S., Studer, R. (eds.) Handbook on Ontologies, pp. 221–243. Springer Verlag (2009)
  • [8]

    Garca-Ramos, S., Otero, A., Fernández-López, M.: OntologyTest: A tool to evaluate ontologies through tests defined by the user. In: Omatu, S., et al. (eds.) 10th International Work-Conference on Artificial Neural Networks, IWANN 2009 Workshops, Proceedings, Part II. LNCS, vol. 5518, pp. 91–98. Springer (2009), salamanca, Spain, June 10-12, 2009.

  • [9] Garcia, A., O’Neill, K., Garcia, L.J., Lord, P., Stevens, R., Corcho, O., Gibson, F.: Developing ontologies within decentralized settings. In: Chen, H., et al. (eds.) Semantic e-Science. Annals of Information Systems 11, pp. 99–139. Springer (2010)
  • [10] Gennari, J.H., Musen, M.A., Fergerson, R.W., Grosso, W.E., Crubézy, M., Eriksson, H., Noy, N.F., Tu, S.W.: The evolution of Protégé: an environment for knowledge-based systems development. International Journal of Human-Computer Studies 58(1), 89–123 (2003)
  • [11] Ghidini, C., Kump, B., Lindstaedt, S., Mabhub, N., Pammer, V., Rospocher, M., Serafini, L.: Moki: The enterprise modelling wiki. In: Proceedings of the 6th Annual European Semantic Web Conference (ESWC2009) (2009), heraklion, Greece, 2009 (demo)
  • [12] Grimm, S., Motik, B.: Closed world reasoning in the semantic web through epistemic operators. In: Grau, B.C., Horrocks, I., Parsia, B., Patel-Schneider, P. (eds.) First OWL: Experiences and Directions Workshop (OWLED’05). CEUR-WS, vol. 188 (2005), galway, Ireland, Nov 11-12, 2005
  • [13] Halpin, T.: Information Modeling and Relational Databases. San Francisco: Morgan Kaufmann Publishers (2001)
  • [14] Janzen, D.S.: Software architecture improvement through test-driven development. In: Companion to 20th ACM SIGPLAN Conference 2005. pp. 240–241. ACM Proceedings (2005)
  • [15] Keet, C.M., d’Amato, C., Khan, Z., Lawrynowicz, A.: Exploring reasoning with the DMOP ontology. In: Bail, S., Glimm, B., Jiménez-Ruiz, E., Matentzoglu, N., Parsia, B., Steigmiller, A. (eds.) 3rd Workshop on Ontology Reasoner Evaluation (ORE’14). CEUR-WS, vol. 1207, pp. 64–70. CEUR-WS (2014), july 13, 2014, Vienna, Austria
  • [16] Keet, C.M., Khan, M.T., Ghidini, C.: Ontology authoring with FORZA. In: Proceedings of the 22nd ACM international conference on Conference on Information & Knowledge Management (CIKM’13). pp. 569–578. ACM proceedings (2013), oct. 27 - Nov. 1, 2013, San Francisco, USA.
  • [17] Kim, T., Park, C., Wu, C.: Mock object models for test driven development. In: Proceedings of the Fourth International Conference on Software Engineering Research, Management and Applications (SERA’06). IEEE Computer Society (2006)
  • [18] Kollia, I., Glimm, B., Horrocks, I.: SPARQL Query Answering over OWL Ontologies. In: Antoniou, G., et al. (eds.) Proceedings of the 8th Extended Semantic Web Conference (ESWC’11). LNCS, vol. 6643, pp. 382–396. Springer (2011), 29 May-2 June, 2011, Heraklion, Crete, Greece
  • [19] Kontokostas, D., Westphal, P., Auer, S., Hellmann, S., Lehmann, J., Cornelissen, R., Zaveri, A.: Test-driven evaluation of linked data quality. In: Proceedings of the 23rd international conference on World Wide Web (WWW’14). pp. 747–758. ACM proceedings (2014)
  • [20] Kumar, S., Bansal, S.: Comparative study of test driven development with traditional techniques. International Journal of Soft Computing and Engineering 3(1), 352–360 (2013)
  • [21] Mackinnon, T., Freeman, S., Craig, P.: Extreme Programming Examined, chap. Endo-testing: unit testing with mock objects, pp. 287–301. Addison-Wesley, Boston, MA (2001)
  • [22] Mehdi, A., Rudolph, S., Grimm, S.: Epistemic querying of OWL knowledge bases. In: Antoniou, G., Grobelnik, M., Simperl, E.P.B., Parsia, B., Plexousakis, D., Leenheer, P.D., Pan, J.Z. (eds.) The Semantic Web: Research and Applications - 8th Extended Semantic Web Conference, ESWC 2011, Heraklion, Crete, Greece, May 29-June 2, 2011, Proceedings, Part I. Lecture Notes in Computer Science, vol. 6643, pp. 397–409. Springer (2011)
  • [23] Parsia, B., Matentzoglu, N., Goncalves, R., Glimm, B., Steigmiller, A.: The OWL Reasoner Evaluation (ORE) 2015 competition report. In: Liebig, T., Fokoue, A. (eds.) Proceedings of the 11th International Workshop on Scalable Semantic Web Knowledge Base Systems (SSWS’15). CEUR-WS, vol. 1457 (2015), bethlehem, USA, Oct 11, 2015.
  • [24] Paschke, A., Schaefermeier, R.: Aspect OntoMaven - aspect-oriented ontology development and configuration with OntoMaven. Tech. Rep. 1507.00212v1, Institute of Computer Science, Free University of Berlin (July 2015), http://arxiv.org/abs/1507.00212
  • [25] Poveda-Villalón, M., Suárez-Figueroa, M.C., Gómez-Pérez, A.: Validating ontologies with OOPS! In: ten Teije, A., et al. (eds.) 18th International Conference on Knowledge Engineering and Knowledge Management (EKAW’12). LNAI, vol. 7603, pp. 267–281. Springer (2012), oct 8-12, Galway, Ireland
  • [26] Presutti, V., Daga, E., et al.: extreme design with content ontology design patterns. In: Proc. of WS on OP’09. CEUR-WS, vol. 516, pp. 83–97 (2009)
  • [27] Presutti, V., Gangemi, A., David, S., de Cea, G.A., Surez-Figueroa, M.C., Montiel-Ponsoda, E., Poveda, M.: A library of ontology design patterns: reusable solutions for collaborative design of networked ontologies. NeOn deliverable D2.5.1, NeOn Project, Institute of Cognitive Sciences and Technologies (CNR) (2008)
  • [28] Ren, Y., Parvizi, A., Mellish, C., Pan, J.Z., van Deemter, K., Stevens, R.: Towards competency question-driven ontology authoring. In: Extended Semantic Web Conference (ESWC’14). LNCS, Springer (2014)
  • [29] Shrivastava, D.P., Jain, R.: Metrics for test case design in test driven development. International Journal of Computer Theory and Engineering, 2(6), 952–956 (2010)
  • [30] Sirin, E., Parsia, B.: SPARQL-DL: SPARQL Query for OWL-DL. In: Proceedings of the Third International Workshop OWL: Experiences and Directions (OWLED’07). CEUR-WS (2007), 6-7 June 2007, Innsbruck, Austria
  • [31] Suárez-Figueroa, M.C., de Cea, G.A., Buil, C., Dellschaft, K., Fernández-Lopez, M., Garcia, A., Gómez-Pérez, A., Herrero, G., Montiel-Ponsoda, E., Sabou, M., Villazon-Terrazas, B., Yufei, Z.: NeOn methodology for building contextualized ontology networks. NeOn Deliverable D5.4.1, NeOn Project (2008)
  • [32] Taneja, K., Zhang, Y., Xie, T.: Moda: Automated test generation for database applications via mock objects. In: Proceedings of the IEEE/ACM international conference on Automated software engineering (ASE’10)). pp. 289–292. ACM proceedings (2010)
  • [33] Tort, A., Olivé, A.: An approach to testing conceptual schemas. Data & Knowledge Engineering 69, 598–618 (2010)
  • [34] Tort, A., Olivé, A., Sancho, M.R.: An approach to test-driven development of conceptual schemas. Data & Knowledge Engineering 70, 1088–1111 (2011)
  • [35] Vrandečić, D., Gangemi, A.: Unit tests for ontologies. In: OTM workshops 2006. LNCS, vol. 4278, pp. 1012–1020. Springer (2006)
  • [36] Warrender, J.D., Lord, P.: How, What and Why to test an ontology. Technical Report 1505.04112, Newcastle University (2015), http://arxiv.org/abs/1505.04112