Comparison of ontology alignment algorithms across single matching task via the McNemar test

03/29/2017
by   Majid Mohammadi, et al.
0

Ontology alignment is widely used to find the correspondences between different ontologies in diverse fields. After discovering the alignment by methods, several performance scores are available to evaluate them. The scores require the produced alignment by a method and the reference alignment containing the underlying actual correspondences of the given ontologies. The current trend in alignment evaluation is to put forward a new score and to compare various alignments by juxtaposing their performance scores. However, it is substantially provocative to select one performance score among others for comparison. On top of that, claiming if one method has a better performance than one another can not be substantiated by solely comparing the scores. In this paper, we propose the statistical procedures which enable us to theoretically favor one method over one another. The McNemar test is considered as a reliable and suitable means for comparing two ontology alignment methods over one matching task. The test applies to a 2 x 2 contingency table which can be constructed in two different ways based on the alignments, each of which has their own merits/pitfalls. The ways of the contingency table construction and various apposite statistics from the McNemar test are elaborated in minute detail. In the case of having more than two alignment methods for comparison, the family-wise error rate is expected to happen. Thus, the ways of preventing such an error are also discussed. A directed graph visualizes the outcome of the McNemar test in the presence of multiple alignment methods. From this graph, it is readily understood if one method is better than one another or if their differences are imperceptible. Our investigation on the methods participated in the anatomy track of OAEI 2016 demonstrates that AML and CroMatcher are the top two methods and DKP-AOM and Alin are the bottom two ones.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/19/2013

Ontology alignment repair through modularization and confidence-based heuristics

Ontology Matching aims to find a set of semantic correspondences, called...
research
08/18/2023

Conversational Ontology Alignment with ChatGPT

This study evaluates the applicability and efficiency of ChatGPT for ont...
research
04/08/2023

Block-regularized 5×2 Cross-validated McNemar's Test for Comparing Two Classification Algorithms

In the task of comparing two classification algorithms, the widely-used ...
research
09/15/2021

Matching with Transformers in MELT

One of the strongest signals for automated matching of ontologies and kn...
research
08/10/2018

Dynamic all scores matrices for LCS score

The problem of aligning two strings A,B in order to determine their simi...
research
04/08/2022

Ontology Matching Through Absolute Orientation of Embedding Spaces

Ontology matching is a core task when creating interoperable and linked ...
research
02/16/2017

Evaluation of Trace Alignment Quality and its Application in Medical Process Mining

Trace alignment algorithms have been used in process mining for discover...

Please sign up or login with your details

Forgot password? Click here to reset