The knowledge graph, a structured knowledge base, represents world’s truth in a form that computer can easily process. As the basis of question answering and knowledge inference, etc., the knowledge graph has received extensive attention from academia and industry.
In recent years, knowledge graph reasoning has made significant progress. There are two main branches, logical reasoning and representation learning, each with its own advantages and disadvantages. Logical reasoning based on the rigorous mathematical foundation is difficult to solve the computational bottleneck of the combinatorial explosion. Knowledge representation learning based on statistics has attracted more attention because of the development of machine learning and deep learning at present, but it is limited by the incompleteness and the scale of the knowledge base.
Usually, each fact of the knowledge graph is represented by a triple , where and are the head entity and the tail entity, respectively, and is the relation between them.
For example, the triple means that Trump’s spouse is Melania, in which Trump is the head entity, the spouse is the relation, and Melania is the tail entity. Semantically, relation is symmetric, shown in figure1, and simultaneously hold.
KGE aims to embed the entities and relations into low-dimensional real vectors, and then learns the representations of them. TransE  is the earliest KGE model and has derived a series of models called Trans series models or Trans models. Most of Trans models based on vector addition calculation, which are difficult to apply well in symmetric relations.
We propose bi-vector models extended the Trans models for symmetric relations. Different from the Trans models using a single vector to represent the entity or relation, We adopt bi-vector to represent symmetric relation. The score functions of the two subvectors are calculated separately. With the increase of training epochs, the two subvectors are separated step by step. And then, models can distinguish the two directions of the symmetric relation
Two benchmark datasets, FB15k-SYM and WN18-SYM construced by us for running bi-vector models on them The experimental results show that our method can effectively improve the triple prediction accuracy of symmetry relations. The main contributions of this paper as follow.
We propose bi-vector models which improve the prediction accuracy of symmetric relations.
The symmetric semantic information of relations is combined with KGE, which is a new research method of knowledge graph reasoning.
We run the model on the extended benchmark datasets and verify the effectiveness and advantages of the models.
2 Related Works
2.1 TransE, TransH and TransD
, the first KGE model proposed, regards relation as the translation from entity to . Entity should be in the nearest neighborhood of . The score function is defined as
Where is usually as norm or norm. TransE can slove 1-1 relations effectively, but it is not suitable for handling 1-n, n-1 and n-n relations.
projects entities and
into the hyperplane which relationlocated. TransH calculates , before calculating score function,
Where is usually as norm. TransH is more accurate than TransE in terms of recognition rate of 1-n, n-1 and n-n relations.
believes that combinations of entities and relations can distinguish the relation more finely. The combination of entity and relation correspondences association matrix . The calculation of score function uses the product of entity and association matrix, form as , . The score function is defined as
Where is usually as norm.
2.2 Other Models
•Translation based methods
. In addition to TransE(H,D) that we have already mentioned, translation based methods cover the following models. TransR  build entity and relation embedding independent spaces, in which, entities , and relation . A projection matrix has been set, and the score funcion is defined as TransSparse  set two separate relation sparse matrices and to deal with the issue of sparse data. The score function is defined as .TransF reduces the cost of calculation of relation projection by modeling subspaces of projection matrices, and the score function is defined as , where ,, and are the corresponding coefficients of and .
•Tensor based methods
. DistMult  adopts a relation-specific diagonal matrix to represents the characteristics of a relation. The score function is a bilinear function, which score of positive triples should be higher than negative triples. HolE  employs circular correlations by holographic to create compositional representations, and has advantages of computation efficiency and representing scalability. RESCAL ComplEX  embed the entities and relation to complex space, then computes loss vaule.
•Other related methods
. SE defines two relation-specific matrices for , i.e. , and defines the score function as . There are many other KGE models try to try to use various embedding methods, such as Neural Tensor Network (NTN) , Semantic Matching Energy (SME), SLM, TransA, lppTransD, etc.
However, these works did not utilize the semantic information of relations properties. We believe that the semantic information of the relations properties are of value and can improve the performance of the KGE models.
In order to overcome the lack of support for symmetric relations in KGE, we made the following efforts. First of all we describe the defects of Trans models in handling symmetric relations, and analyze the causes of it. Then, we propose three new models that extends the Trans models to improve the performance of handling symmetry relations in KGE, which are named TransE-SYM, TransH-SYM and TransD-SYM. Finally, we give the definition of the loss functions for these models.
3.1 Problems and causes
Knowledge graph can be represented as a set of ordered triples of entities and relations. Each triple in Knowledge graph is essentially a binary relation, which have the properties of symmetry, anti-symmetric, reflexive, anti-reflexive and transitive properties. This paper focuses on the relation’s properties of symmetry. In graph, symmetric relation have two directed edges in opposite directions.
KGE represents each relation, including symmetric relation, as a low-dimensional real vector. However, a single vector cannot represent two opposite directions.
We take TransE as an example to illustrate the problem of symmetric relations. TransE learns the embedding feature from equation when triplets holds. TransE’s scoring function is defined as . When the function , it means .
Assuming that there is a symmetric relation and triple in , then , ie . Since is symmetric, then the symmetric triple should hold too, satisfying , ie .
Obviously, if both and are correct, if and only if is an additive identity of vector, ie , the conclusion contradicts with the conditions of TransE model.
Taking the symmetric relation as an example, shown in figure1. When the fact holds, the fact holds too. let , and denote entities Melania, Trump and relation spouse, respectively. Then,
According to the KGE preset, the relations should be a non-zero real vector, and Equation (6) contradicts with the condition. The root cause of the above problem is that the symmetric relation is represented by single vector, and the single vector cannot express semantic bifurcation of symmetric relation.
3.2 Our Method
Aiming at these problems, bi-vector models for symmetric relation are presented in this study.
Knowledge graph , , Where and are entities set and relations set, respectively.
Symmetric relation , if and are entities of knowledge graph , is the relation of , and , , then relation is symmetric relation.
Different from most of KGE models, which represent entities and relations as single vector, we represent the symmetric relation as a bi-vector with two subvectors, and . Then, in each epoch of learning, the score functions of the two subvectors are calculated, and the better score is selected as the current result. Let be the score function of the Trans series model, as show in Equation(7)
We have extended three different Trans models, which differ in their respective score functions. In TransE, score function is , where is L1 norm or L2 norm, and the score functions of subvectors are shown as Equation array(8),
and should be substituted into the following loss function,
where denotes the margin of hyperplane, and denotes . Similarly, the score function of the TransH model is shown in Equation array (10).
The score function of the TransH model is shown in Equation array (11).
The loss functions of them are calculated according to Equation (9).
4 Experiments and results
4.1 Dataset analysis and preprocessing
In this study, we compared and analyzed the commonly used knowledge graph embedding benchmark data sets FB15k, FB15k-237, FB13, WN18, WN11 and WN18RR. FB15k, FB15k-237 and FB13 are extracted from Freebase, which is a large-scale common sense knowledge base provided the general facts of the world. Freebase was acquired by Google and is still under maintenance. WN18, WN11 and WN18R aextract from WordNet  and provide semantic knowledge of words.
We count the ratio of the symmetric relations in the data set shown in the table 1. It can be seen that the proportion of symmetric data of the WN18 and FB15k data set are relatively high.
The proportion of symmetric data for relation is denoted as by the paper. We regard as symmetric relation When exceeds the threshold111In this paper, the threshold is set to 0.5..
As shown in the table2, in WN18, the relation has 1139 triples, of which 1060 are symmetric triples, and the ratio of symmetric triples is about 0.93. Semantically, the relation is the meaning of verb grouping, which is obviously a symmetric relation. From the perspective of data distribution, the symmetry rate of the relation is 0.93, and we believe it is symmetrical.
In order to simplify the problem, in this paper, symmetry is only judged by data distribution.We complement the missing symmetric triples in dataset of the symmetric relation. A more formal description is, if relation in knowledge graph is symmetric, for , if and then .
In order to show the superiority of our models, we compare the following benchmark KGE models.
is the most widely used KGE model, also the earliest proposed KGE model.
projects h and t to the hyperplane where r located, to solve the relations of 1-n, n-1 and n-n.
uses the entity-relation matrix to obtain a more fine-grained distinction of realtion.
4.3 Verification problem
In order to verify the problem of the Trans models described in Section 3.1, We have designed the following experiments, the steps are as follows.
Training Trans models. We train the TransE, TransH and TransD models on the datasets which are completed symmetric triples in Section 4.1.
Constructing test dataset.We randomly selected symmetric relations and entities in FB15k and WN18 to construct test sets. Each test set contains 10,000 symmetric triples named FB15k-test-circle and WN18-test-circle. The form of triples in test sets is , where and are respectively symmetric relation and any entity. The triple example is as follows,
Experimental results. According to Section 3.1, if the symmetric triple is true, the relation tends to zero. We run the test sets on models and the experimental results are shown in Table 3. Almost all randomly generated triples is true. These models completely fail in dealing with all of symmetric relations.
|Model||Train Dataset||Test Dataset||MR||MRR||H10||H3||H1|
4.4 Result of Experiment.
Three bi-vector Trans models named TransE-SYM, TransH-SYM and TransD-SYM proposed by us. Experimental code implementation reference open source project OpenKE. These models run on datasets completed symmetric relation and get good results. The experimental results are shown in Table 4. Bi-vector models are superior to the original model in indicators of the link prediction task.
This paper introduces symmetry semantics into KGE models, and points out the defect of the state-of-the-art KGE models learning symmetric relations. Bi-vector models proposed by us can improve the situation of low recognition rate of symmetric relations in Trans models.
-  Antoine Bordes, Nicolas Usunier, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In NIPS.
-  Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge graph embedding by translating on hyperplanes. In AAAI.
-  Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Knowledge graph embedding via dynamic mapping matrix. In ACL.
-  Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning entity and relation embeddings for knowledge graph completion. In AAAI.
Guoliang Ji, Kang Liu, Shizhu He, and Jun Zhao. (2016). Knowledge graph completion with adaptive sparse transfer matrix. Thirtieth Aaai Conference on Artificial Intelligence.
-  Bishan Yang, Wentau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding entities and relations for learning and inference in knowledge bases. In ICLR.
-  Maximilian Nickel, Lorenzo Rosasco, and Tomaso Poggio. 2016. Holographic embeddings of knowledge graphs. In AAAI.
-  Nickel, M., Tresp, V., Kriegel, H. P. (2011, June). A Three-Way Model for Collective Learning on Multi-Relational Data. In ICML (Vol. 11, pp. 809-816).
-  Johannes Welbl, Sebastian Riedel, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In ICML.
-  Antoine Bordes, Jason Weston, Ronan Collobert, and Yoshua Bengio. 2012. Learning structured embeddings of knowledge bases. In AAAI.
-  Richard Socher, Danqi Chen, Christopher D. Manning, and Andrew Y. Ng. 2013. Reasoning with neural tensor networks for knowledge base completion. In NIPS.
-  Antoine Bordes, Xavier Glorot, and Jason Weston. 2012. Joint learning of words and meaning representations for open-text semantic parsing. In International Conference on Artificial Intelligence and Statistics.
-  Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In SIGMOD.
-  George A. Miller. 1994. Wordnet: a lexical database for english. In The Workshop on Human Language Technology.
-  Han, Xu and Cao, Shulin and Lv, Xin and Lin, Yankai and Liu, Zhiyuan and Sun, Maosong and Li, Juanzi. 2018. OpenKE: An Open Toolkit for Knowledge Embedding. In Proceedings of EMNLP