Knowledge Graphs are increasingly used as means to represent knowledge. Due to their versatile means of representation, they can be used to integrate different heterogeneous data sources, both within as well as across organizations. [galkin2016enterprise, gomez2017enterprise]
Besides such domain-specific knowledge graphs which are typically developed for specific domains and/or use cases, there are also public, cross-domain knowledge graphs encoding common knowledge, such as DBpedia, Wikidata, or YAGO. [ringler2017one] Such knowledge graphs may be used, e.g., for automatically enriching data with background knowledge to be used in knowledge-intensive downstream applications. [ristoski2016semantic] In particular for the case of explainable AI, knowledge graphs can be used as additional input to the AI algorithm, as a means to support interpretation of the results, or both. [lecue2019role]
Since Google coined the term Knowledge Graph for marketing purposes, it has subsequently been used in the scientific literature as well. The slogan by which Google announced KGs was Things, not Strings.111https://www.blog.google/products/search/introducing-knowledge-graph-things-not/ The idea of that slogan is: while strings are often ambiguous, knowledge graphs consist of disambiguated entities, so that entities of the same name can be told apart more easily. Nowadays, almost all companies processing large amounts of heterogeneous data use knowledge graphs as a means of representation, including, but not limited to IBM, Microsoft, Facebook or Ebay. [noy2019industry].
There are quite a few different definitions for knowledge graphs. [ehrlinger2016towards]. Typically, a knowledge graph
mainly describes real world entities and their interrelations, organized in a graph.
defines possible classes and relations of entities in a schema.
allows for potentially interrelating arbitrary entities with each other.
covers various topical domains. [paulheim2017kg]
In this chapter, we provide an overview of publicly available, cross-domain knowledge graphs on the Web. We discuss the techniques used to create those knowledge graphs and provide an in-depth comparison in terms of size, level of detail, contents, and overlap.
There are different techniques for creating knowledge graphs. The most common ones are (1) manual curation, (2) creation from (semi) structured sources, and (3) creation from unstructured sources. Some knowledge graphs also use a mix of those techniques.
2.1 Manual Curation
Cyc [lenat1995cyc] is one of the oldest knowledge graphs; the Cyc project dates back to the 1990s. Cyc was created along with its own language (CycL), which provides a large degree of formalization.
While Cyc was developed by a comparatively small group of experts, the idea of Freebase [pellissier2016freebase] was to establish a large community of volunteers, compared to Wikipedia. To that end, the schema of Freebase was kept fairly simple to lower the entrance barrier as much as possible. Freebase was acquired by Google in 2010 and shut down in 2014.
Wikidata [vrandevcic2014wikidata] also uses on a crowd editing approach. In contrast to Cyc and Freebase, Wikidata also imports entire whole large datasets, such as several national libraries’ bibliographies. Porting the data from Freebase to Wikidata is also a long standing goal [pellissier2016freebase].
Curating a knowledge graph manually can be a large effort. The total cost of development for Cyc have been estimated as 120 Million USD.222http://www.ttivanguard.com/conference/Napa2017/4-Lenat.pdf This corresponds to a total cost of 2-6 USD per single axiom in Cyc. [paulheim2018much].
2.2 Creation from (Semi) Structured Sources
A more efficient way of knowledge graph creation is the use of structured or semi structured sources. Wikipedia is a commonly used starting point for knowledge graphs such as DBpedia [LehmannDBpedia] and YAGO [SuchanekYAGO].
DBpedia mainly uses infoboxes in Wikipedia. Those are manually mapped to a pre-defined ontology; the mapping is crowd sourced using a Wiki and a community of volunteers. Given those mappings, the DBpedia Extraction Framework creates a graph in which each page in Wikipedia becomes an entity, and all values and links in an infobox become attributes and edges in the graph.
uses a similar process, but classifies instances based on the category structure and WordNet[miller1995wordnet] instead of infoboxes. YAGO integrates various language editions of Wikipedia into a single graph and represents temporal facts with meta-level statements, i.e., RDF reification.
CaLiGraph also uses information in categories, but aims at converting them into formal axioms using DBpedia as supervision [heist2019uncovering]. Moreover, instances from Wikipedia list pages are considered for populating the knowledge graph [paulheim2013extending]. The result is a knowledge graph which is not only richly populated on the instance level, but also has a large number of defining axioms for classes [heist2020caligraph].
A similar approach, i.e., the combination of information in Wikipedia and WordNet, is used by BabelNet [navigli2012babelnet]. The main purpose of BabelNet is the collection of synonyms and translations in various languages, so that this knowledge graph is particularly well suited for supporting multi-language applications. Similarly, ConceptNet [speer2012representing] collects synonyms and translations in various languages, integrating multiple third party knowledge graphs itself.
DBkWik [hertling2018dbkwik] uses the same codebase as DBpedia, but applies it to a multitude of Wikis. This leads to a graph which has a larger coverage and level of detail for many long tail entities, and is highly complementary to DBpedia. However, the absence of a central ontology and mappings, as well as the existence of duplicates across Wikis, which might not be trivial to detect, imposes a number of challenges not present in DBpedia.
Another source of structured data is the structured annotations in Web pages using techniques such as RDFa, Microdata, and Microformats [meusel2014webdatacommons]. While the pure collection of those could, in theory, already be considered a knowledge graph, that graph would be rather disconnected and consist of a plethora of small, unconnected components [paulheim2015adoption] and would require additional cleanup for compensating irregular use of the underlying schemas and shortcomings in the extraction [meusel2015heuristics]. A consolidated version of this data into a more connected knowledge graph has been published under the name VoldemortKG [tonon2016voldemortkg].
2.3 Creation from Unstructured Sources
The extraction of a knowledge graph from semi structured sources is considered more easy than from the extraction from unstructured sources. However, there is much more information in unstructured sources (such as text). Therefore, extracting knowledge from unstructured sources has also been proposed.
NELL [carlson2010coupled] is an example for extracting a knowledge graph from free text. NELL was originally trained with a few seed examples and continuously runs an iterative coupled learning process. In each iteration, facts are used to learn textual patterns to detect those facts, and patterns learned in previous iterations are used to extract new facts, which serve as training examples in later iterations. To improve the quality, NELL has introduced a feedback loop incorporating occasional human feedback.
WebIsA [seitner2016large] also extracts facts from free text, but focuses on the creation of a large-scale taxonomy. For each extracted fact, rich metadata are collected, including the sources, the original sentences, and the patterns used in the extraction of a particular fact. Those metadata are exploited for computing a confidence score for each fact. [hertling2017webisalod].
3 Comparison of Knowledge Graphs
Whenever a knowledge graph is to be used in an application, it is important to determine which knowledge graph is best suitable for an application at hand. The knowledge graphs mentioned above differ in their content, their level of detail, etc. Hence, in this chapter, we will discuss several characteristics of knowledge graphs and provide insights into the differences between them.
3.1 General Metrics
The most straightforward metrics to be used consider the mere amount of information contained in a knowledge graph. Measures that may be used include:
The number of instances in a graph
The number of assertions (or edges between entities)
The average and median linkage degree (i.e.: how many assertions per entity does the graph contain?)
As for using a knowledge graph in an XAI system, these metrics hint at the utility – the more information about the domain at hand is present (i.e., the more instances are represented in the knowledge graph and the more detailed that information is), the more can an XAI application benefit in providing better results or better interpretations.
Another set of metrics can be defined for the schema or ontology level of a knowledge graph:
The number of classes defined in the schema
The number of relations defined in the schema
The average depth and width (branching factor) of the class hierarchy333While this could also be done for the property hierarchy, extensive property hierarchies are rather rare in common knowledge graphs.
The complexity of the schema
While the instance-based metrics focus more on the coverage of a domain in a knowledge graph, these schema-level metrics provide information about the richness and formality of that knowledge. They determine which techniques to use – e.g., while more formal, very complex ontologies will call for using ontology reasoning, light-weight, but large-scale ontologies will be better exploited by statistical and distributional approaches.
Table 1 depicts those metrics for some of the knowledge graphs discussed above. ConceptNet and WebIsA are not included, since they do not distinguish a schema and instance level (i.e., there is no specific distinction between a class and an instance), which does not allow for computing those metrics meaningfully. For Cyc, which is only available as a commercial product today, we used the free version OpenCyc, which has been available until 2017.444It is still available, e.g., at https://github.com/asanchez75/opencyc
|Avg. linking degree||21.30||48.26||6.38||0.00|
|Median ingoing edges||0||0||0||0|
|Median outgoing edges||30||95||10||9|
|Avg. depth of class tree||3.51||6.61||6.43||4.11|
|Avg. branching factor of class tree||4.53||8.48||36.48||71.0|
|Avg. linking degree||3.34||6.72||1.48||0|
|Median ingoing edges||0||0||0||0|
|Median outgoing edges||3||0||1||5|
|Avg. depth of class tree||5.58||3.13||4.74||3.17|
|Avg. branching factor of class tree||5.62||6.37||4.81||5.40|
From those metrics, it can be observed that the KGs differ in size by several orders of magnitude. The sizes range from 50,000 instances (and Voldemort) to 50 million instances (for Wikidata), so the latter is larger by a factor of 1,000. The same holds for assertions. Concerning the linkage degree, YAGO is much richer linked than the other graphs.
Figure 1 shows an overview of the knowledge graphs considered. We follow the conventions of the Linked Open Data Cloud diagrams555https://www.lod-cloud.net/ [schmachtenberg2014adoption], which are used to depict linked datasets and their connections. In those diagrams, the size of the circles is proportional to the number of instances, and the strength of the connecting lines is proportional to the number of links.
The knowledge graphs also differ strongly in the characteristics of their schema. DBpedia and NELL have comparably small schemas, while Wikidata and BabelNet build deep and detailed taxonomies. For example, while NELL does not define detailed subclasses for Scientist666http://rtw.ml.cmu.edu/rtw/kbbrowser/pred:scientist, DBpedia defines four subclasses777http://dbpedia.org/ontology/Scientist, Wikidata has more than 600888https://www.wikidata.org/wiki/Q15976092 and CaLiGraph almost 2,000999http://caligraph.org/ontology/Scientist, including detailed classes such as sickle-cell disease researcher or loop quantum gravity researcher. Voldemort, on the other hand, reuses the schema.org ontology, which is comparably small [meusel2015web].
Looking at the complexity, it is not much of a surprise that Cyc, originating in classic AI research and strongly building on logical rules [buchanan2005very, lenat1995cyc], has the highest complexity. Wikidata, BabelNet, and Voldemort have only little complexity, the other graphs are somewhere inbetween.
The knowledge graphs do not only differ in their size and level of detail, but also in their contents. The most straightforward way to assess the content focus of a knowledge graph is to look at the size of its classes. Figures 2-9 show graphic depictions of those class sizes. The diagrams were created starting from the most abstract class and following the class hierarchy to the largest respective subclasses.
At first glance, the figures reveal differences in the development of the taxonomies. While Cyc builds a formal ontology with very abstract top level categories such as partially intangible thing or thing that exists in time, the more pragmatic classification in DBpedia and Voldemort (the latter using schema.org as an ontology) has top level classes such as Place or Person. The reason for these differences lies in the origins of the respective knowledge graphs: While Cyc’s classification was created by AI researchers, the ontology in DBpedia is the result of a crowdsourcing process [paulheim2018much]. The same holds for schema.org, which is a pragmatic effort of a consortium of search engine developers.
Moreover, the diagrams reveal some differences in the contents. The main focus of DBpedia is on persons (and their careers), as well as places, works, and species. Wikidata also has a strong focus on works (mainly due to the import of entire bibliographic datasets), while Cyc, BabelNet and NELL show a more diverse distribution.
3.3 Looking into Details
To obtain deeper insights which classes are more prominent in which KGs, and, ultimately, which KGs are suitable for building explainable AI system in a specific domain, it is useful to not only look at the number of instances, but also the level of detail in which those instances are represented (i.e., the linkage degree and number of assertions per instance).
Table 2 depicts such a detailed view for ten prominent classes:
Populated place (city, country, etc.)
Uninhabited place (mountain, lake, etc.)
Work (book, movie, etc.)
|Class||Instances||Avg. Deg.||Med-in||Med-out||Instances||Avg. Deg.||Med-in||Med-out||Instances||Avg. Deg.||Med-in||Med-out|
|Class||Instances||Avg. Deg.||Med-in||Med-out||Instances||Avg. Deg.||Med-in||Med-out||Instances||Avg. Deg.||Med-in||Med-out|
|Class||Instances||Avg. Deg.||Med-in||Med-out||Instances||Avg. Deg.||Med-in||Med-out|
The global trend observed in this table is that Wikidata has the largest number of instances in most of the classes, while YAGO has the largest level of detail. However, there are differences from class to class. While Wikidata has a large number of works, YAGO is a good source of events. NELL often has fewer instances, but a larger level of detail, which can be explained by its focus on more prominent instances.
The contrast of the average and the median degree also reveals a few differences. For example, BabelNet contains a similar amount of instances as DBpedia for some classes, e.g., uninhabited places or works. While the average linkage degree is higher in DBpedia, the median is higher in BabelNet. This hints at a more uneven distribution of information in DBpedia, while BabelNet has a more constant distribution of statements per instance.
4 Linkage and Overlap of Knowledge Graphs
Since knowledge graphs differ so strongly in size, coverage, and level of detail, combining information from multiple KGs for implementing one application is often beneficial. To estimate the value of such a combination, we determine the overlap of the knowledge graphs first.
As shown in Fig. 1, many KGs contain explicit interlinks. Those links, usually in the form of owl:sameAs links, express that entities in two KGs are the same (or, more precisely: that they refer to the same real world entity) [halpin2010owl]. In other cases, such links can be generated indirectly, e.g., if a knowledge graph contains links to Wikipedia pages, which can be easily mapped to entities in DBpedia and YAGO.
Even if those links provide a first hint at the overlap of KGs, and further links can be found by exploiting the transitivity of the owl:sameAs property [beek2018sameas], they do not provide a complete picture. Due to the open world assumption, which holds for KG interlinks as well, there might always be more links than the one which are explicitly or implicitly provided by the KGs.
In order to estimate the actual number of interlinks, we use a method first discussed in [ringler2017one]
, which builds on a set of existing links and heuristic link discovery:
We use different heuristics to discover links between two KGs automatically, e.g., different string similarity measures [ferrara2011data, nentwig2017survey].
Based on the existing, incomplete set of interlinks, we measure recall and precision of the individual heuristics [ritze2011towards].
With the help of those recall and precision figures, we can estimate the actual number of interlinks. After repeating the procedure with multiple heuristics, we can use the average of those estimations.
Given that the actual number of links is (which is unknown), the number of links found by a heuristic is , and that the number of correct links in is , recall and precision are defined as
By resolving both to and combining the equations, we can estimate as
Thus, we can obtain an estimate for given , , and . A more intuitive interpretation of the last equation is that is a measure of how strongly the heuristic overestimates the number of actual interlinks (thus, is reduced by multiplication with ), and is a measure of how strongly the heuristic underestimates the number of actual interlinks (thus, is divided by ).
In [ringler2017one], we have shown that across different heuristics, although varies a lot, the estimate is fairly stable. For producing the estimates in this chapter, we have used the following heuristics: string equality, scaled Levenshtein (thresholds 0.8, 0.9, and 1.0), Jaccard (0.6, 0.8, and 1.0), Jaro (0.9, 0.95, and 1.0), JaroWinkler (0.9, 0.95, and 1.0), and MongeElkan (0.9, 0.95, and 1.0). The estimated overlap reported is the average estimate computed using these 16 metrics.
To analyze the benefit of the combination of different KGs, we depict the number of estimated links both in relation to (a) the entities existing in the larger of the two KGs (Fig. 10) as well as (b) in relation to the links that exist explicitly or implicitly (Fig. 11). From (a), we can estimate the amount of gain in knowledge of combining two KGs (i.e., if only a small fraction of one KG is also contained in the other and vice versa, such a combination adds a lot of information). From (b), we can get insights into whether the set of existing links is sufficient for such a combination or not.
Fig. 10 shows that in most cases, the larger of two knowledge graphs contains most of the entities of a smaller one, i.e., its set of entities of a class in larger KG is usually a superset of that set in the smaller one. For example, as depicted in table 2, Wikidata contains about twice as many persons as DBpedia and YAGO. A value close to 0 for the overlap implies that DBpedia and YAGO contain almost no persons which are not contained in Wikidata. In conclusion, combining Wikidata with DBpedia or YAGO for a better coverage of the Person class would not be beneficial.
Notable exceptions are BabelNet and CaLiGraph, which often contain complementary instances. For example, DBpedia, BabelNet and CaLiGraph contain 1.2M, 2.4M, and 1.9M instances of the class Person, respectively, while DBpedia and BabelNet together are estimated to 2.9M, and all three together are estimated to contain even 3.9M instances of the class Person. The reasons for the high complementary of DBpedia/YAGO, BabelNet and CaLiGraph are their sources (only English Wikipedia vs. multiple language editions) and extraction mechanisms (especially the extraction from list pages in CaLiGraph, which leads to a larger number of instances overall).
Fig. 11 shows that the linkage between DBpedia, YAGO, BabelNet and CaLiGraph is mostly complete (i.e., most of the common instances are also explicitly linked). Since they are all generated from Wikipedia with different means, this is not much surprising. On the other hand, Nell, OpenCyc, and Voldemort have a much lower degree of linkage. This shows that links between KGs are only complete where they are trivial to create, and combining different knowledge graphs otherwise requires efforts in improving the interlinking as a preliminary step.
5 Conclusion and Outlook
In this chapter, we have given an overview of publicly available, cross-domain knowledge graphs on the Web. We have compared them according to different metrics which might be helpful to implement an explainable AI project in a given domain.
Besides the metrics used for this comparison, there are quite a few more which help in the selection and assessment of a given KG. For example, data quality in KGs has not been considered in this chapter, since there are already quite elaborate surveys covering this aspect [farber2016linked, zaveri2016quality].
So far, we have measured the overlap of knowledge graphs only based on entities. Another helpful metric would be the overlap on the statement level. Even if two knowledge graphs cover the same entity, the information they contain about that entity might still be complementary. For example, for the entity University of Mannheim, DBpedia has the exact number of undergraduate students, PhD students, etc.101010http://dbpedia.org/page/University_of_Mannheim, while Wikidata lists all faculties111111https://www.wikidata.org/wiki/Q317070 and can provide a list of researchers employed at the university121212https://w.wiki/7UU. The density of information differs as well: while YAGO lists 3 alumni of the University of Mannheim131313https://bit.ly/2U4wL0A, DBpedia lists 11 and Wikidata even 85 alumni141414https://w.wiki/7UV. Even contradicting information can be found [bryl2014interlinking]: for example, DBpedia and Wikidata provide a different number of students and Wikidata and YAGO provide different founding dates of the University of Mannheim.
Developing cross-domain knowledge graphs is an active field of research, and new developments emerge every once in a while. They differ in the data they use and/or the method of extraction:
DBkWik [hertling2018dbkwik, hertling2019dbkwik, hofmann2017dbkwik] uses the extraction mechanism of DBpedia and applies it to a multitude of Wikis. The intermediate result is a collection of a few thousand isolated knowledge graphs, which have to be integrated into a coherent joint knowledge graph.
Chaudron [subercaze2017chaudron] uses Wikipedia as a source and focuses on quantifiable values (e.g., sizes, weights, etc.). Besides the mere extraction, Chaudron uses sophisticated methods for recognizing and converting units of measurement.
The Linked Hypernym Dataset (LHD) [kliegr2016lhd], like the aforementioned WebIsALOD, focuses on the extraction of a hypernym graph. It uses a deep linguistic analysis of the first paragraph in Wikipedia.
ClaimsKG [tchechmedjiev2019claimskg] extracts claims from fact checking Web pages, such as politifact, and interlinks them with other knowledge graphs such as DBpedia, which also allows for finding related claims.
The methods discussed in this chapter can be used to assess those emerging knowledge graphs and discuss their added value over existing ones. So, for example, for the above mentioned DBkWik, we have shown that it is highly complimentary to DBpedia: 95% of all entities in DBkWik are not contained in DBpedia and vice versa.
In summary, knowledge graphs are a useful ingredient to XAI systems, as they provide ready-to-use cross-domain knowledge. With this chapter, we have given an overview of existing knowledge graphs on the Web, and some guidelines on picking one or more such graphs to build an application for a task at hand.
Appendix A Data sources for the comparison of knowledge graphs
For the comparison of the knowledge graphs, the following data sources have been used:
BabelNet Version 3.6 is not publicly available for download, but has been provided by the developers upon request.