Contemporary information extraction from text, relation inference in knowledge graphs (KGs), and question answering (QA) are informed by continuous representations of words, entities, types and relations. Faced with the query “Name scientists who played the violin,” and having collected candidate response entities, a QA system will generally want to verify if a candidate is a scientist. Testing if or , where is an entity and are types, is therefore a critical requirement. Unlike Albert Einstein, lesser-known candidates may not be registered in knowledge graphs, and we may need to assign a confidence score of belongingness to a target type.
A common recipe for inferring general relations between entities is to fit suitable vectors to each of them, and to train a network to input query vectors and predict presence or absence of the probed relationship. A key question has been whether types merit a special representation, different from the generic devices that represent KG relations, because of their special properties. Two types may be disjoint, overlapping, or one may contain the other. Containment is transitive.
Compared to the vast array of entity-relation representations available (Bordes et al., 2013; Nickel, 2013; Nickel et al., 2016; Trouillon et al., 2016; Xie et al., 2017), few proposals exist (Vilnis and McCallum, 2014; Vendrov et al., 2015; Jameel and Schockaert, 2016) for representing types to satisfy their specific requirements. Of these, only order embedding (OE) by Vendrov et al. (2015) directly enforces transitivity by modeling it as elementwise vector dominance.
We make three contributions. First, we present a significant improvement to the OE loss objective. Second, we generalize OE to rectangle
embeddings for types: types and entities are represented by (hyper-)rectangles and points respectively. Ideally, type rectangles contain subtype rectangles and entity instance points. Rather than invoke established neural gadgets as black boxes, we introduce constraints and loss functions in a transparent manner, suited to the geometric constraints induced by the task at hand. Third, we remove a limitation in the training and evaluation protocol ofVendrov et al. (2015), and propose a sound alternative. Experiments using synsets from the WordNet noun hierarchy (same as Vendrov et al. (2015)) show the benefits of our new formulations. Our code will be available111https://gitlab.com/soumen.chakrabarti/rectangle.
2. Related work
Words and entities222Also see wiki2vec https://github.com/idio/wiki2vec are usually embedded as points or rays from the origin (Mikolov et al., 2013; Pennington et al., 2014; Yamada et al., 2017). It is well appreciated that relations need more sophisticated representation (Bordes et al., 2013; Nickel, 2013; Nickel et al., 2016; Trouillon et al., 2016; Xie et al., 2017), but types seem to have fallen by the wayside, in relative terms. Vilnis and McCallum (2014) pioneered a Gaussian density representation for words, to model hypernymy via the asymmetric KL divergence as an inference gadget. Items are represented by Gaussian densities (with suitable mean and covariance parameters). If we want low . Normalized densities with unit mass seem inappropriate for types with diverse population sizes. Athiwaratkun and Wilson (2018) have used a thresholded divergence . However, modeling asymmetry does not, in itself, enforce transitivity. Neither is anti-symmetry modeled. Jameel and Schockaert (2016) proposed using subspaces to represent types. They do not address type hierarchies or transitive containment. Recently, Nickel and Kiela (2017) introduced an elegant hyperbolic geometry to represent types, but moving away from Euclidean space can complicate the use of such embeddings in downstream applications, in conjunction with conventional word embeddings. Vendrov et al. (2015) proposed a simpler mechanism: embed each type to vector , and, if , then require , where is elementwise. I.e., must dominate . OE was found better at modeling hypernymy than Gaussian embeddings. In OE, types are open cones with infinite volume, which complicates representing various intersections.
3. : OE with improved loss objective
In what follows, we use the partial order to unify and for notational simplicity. If , OE required . OE defines , which is 0 iff . Given labeled positive instances and negative instances , the overall loss is the sum of two parts:
where is a tuned additive margin. The intuition is that when , we want . There are two limitations to the above loss definitions. First,
is too sensitive to outliers. This is readily remedied by redefiningusing L1 norm, as
is the hinge/ReLU operator. But the semantics ofare wrong: we are needlessly encouraging all dimensions to violate dominance, whereas violation in just one dimension would have been enough.
Specifically, for , loss should be zero if for any . Accordingly, we redefine
so that the loss is zero if dominance fails in at least one dimension. To balance this form in case of positive instances, we redefine
so that the loss is zero only if dominance holds in all dimensions.
The unbounded hinge losses above mean a few outliers can hijack the aggregate losses and . Moreover, the absence of a SVM-like geometric margin (as distinct from the loss margin above) also complicates separating and
cases confidently. Our final design introduces a nonlinearity (sigmoid function) to normalize per-instance losses, additive margin
and a standard stiffness hyperparameter.
(Obviously the ‘’ terms are immaterial for optimization, but bring the loss expression to zero when there are no constraint violations.)
4. Rectangle embeddings
Despite its novelty and elegance, OE has some conceptual limitations. A type with embedding is the infinite axis-aligned open convex cone with its apex at . Thus, types cannot “turn off” dimensions, all pairs of types intersect (although the intersection may be unpopulated), and all types have the same infinite measure, irrespective of their training population sizes.
We propose to represent each type by a hyper-rectangle (hereafter, just ‘rectangle’), a natural generalization of OE cones. A rectangle is convex, bounded and can have collapsed dimensions (i.e., with zero width). Obviously, rectangles can be positioned to be disjoint, and their sizes can give some indication of the number of known instances of corresponding types. Containment of one rectangle in another is transitive by construction, just like OE. Entities remain represented as points (or infinitesimal rectangles for uniform notation).
Each type or entity is represented by a base vector , as well as a nonnegative width vector , so that in dimension , the rectangle has extent . Informally, the rectangle representing is bounded by “lower left corner” and “upper right corner” . For entities, . For types, are regularized with a L2 penalty. The rectangles are allowed to float around freely, so are not regularized.
If or , the rectangle representing must be contained in the rectangle representing . Let the violation in the th dimension be
Then the loss expression for positive instances is
This ensures that the loss is proportional to the largest violating margin and that the loss is zero if the rectangle of is contained in the rectangle of . Analogously, we define
As in , we can add margin, stiffness, and nonlinearity to rectangles, and get
5. Training and evaluation protocols
Because the training and evaluation instances are tuple samples from a single (partially observed) partial order, great care is needed in designing the training, development and testing folds. To use unambiguous short subscripts, we call them learn, dev and eval folds, each with positive and negative instances . Let be the raw set of tuples ( or ). The transitive closure (TC) of , denoted , includes all tuples implied by via transitivity.
5.1. OE protocol
Vendrov et al. (2015) followed this protocol:
Sample positive eval fold .
Sample positive learn fold .
Sample positive dev fold .
Generate negative eval, learning and dev folds, & (see below).
A negative tuple is generated by taking a positive tuple and perturbing either of them randomly to or , where are sampled uniformly at random. In OE negative folds were the same size as positive folds.
The WordNet (Miller et al., 1993) hypernymy data set used by Vendrov et al. (2015) has and . and , sampled from , had only 4000 tuples each. All remaining tuples were in the learn fold. Vendrov et al. (2015) freely admit that “the majority of test set edges can be inferred simply by applying transitivity, giving [them] a strong baseline.” They reported that the TC baseline gave a 0/1 accuracy of 88.2%, Gaussian embeddings (Vilnis and McCallum, 2014) was at 86.6%, and OE at 90.6%.
Instead of 0/1 accuracy, Figure 1 shows the more robust F1 score on test instances achieved by transitive closure and OE, as the size of training data is varied. Vendrov et al. (2015) reported accuracy near the right end of the scale, where OE has little to offer beyond TC. In fact, OE does show significant lift beyond TC when training data is scarce. As we shall see, even with ample training data, and rectangle embeddings improve on OE.
5.2. Sanitized OE protocol
Clearly, evaluation results must be reported separately for instances that cannot be trivially inferred via TC, where the algorithm needs discover a suitable geometry from the combinatorial structure of beyond mere reachability. To this end, we propose the following sanitized protocol.
Sample positive learn fold .
Negative learn fold of size is generated by repeating as needed:
Perturb one of or to get .
If , discard.
Sample positive dev fold .
Discard from if or not found in (explained below).
Sample positive eval fold .
Discard elements from using the same protocol used to discard elements from .
Generate negative dev and eval folds, and , using the same protocol used to generate from .
An entity or type never encountered in the learn fold cannot be embedded meaningfully (unless corpus information is harnessed, see Section 7), so it is pointless to include in dev or eval folds instances that contain such entities or types. Such sampled instances are discarded. To fill folds up to desired sizes, we repeatedly sample pairs until we can retain enough instances.
We prepare our data set similar to Vendrov et al. (2015). WordNet (Miller et al., 1993) gives 82115 (hypernym, hyponym) pairs which we use as directed edges to construct our KG. The WordNet noun hierarchy is prepared by experts, and is also at the heart of other type systems (Suchanek et al., 2007; Murty et al., 2017) used in KG completion and information extraction. We augment the KG by computing its transitive closure, which increases the edge count to 838073. Then we use the two protocols in Section 5 to create training, dev and test folds. The sanitized protocol produces 679241 positive and 679241 negative training instances, 4393 positive and 4393 negative dev instances, and 4316 positive and 4316 negative test instances. These sizes are close to those of Vendrov et al. (2015).
Code and hyperparameter details:
OE and our enhancements,
and rectangle embeddings, were coded in Tensorflow with Adam’s optimizer. Hyperparameters, such as batch size (500), initial learning rate (0.1), marginand stiffness
, were tuned using the dev fold. Optimization was stopped if the loss on the dev fold did not improve more than 0.1% for 20 consecutive epochs. All types and entities were embedded todimensions.
Vendrov et al. (2015) reported only microaveraged 0/1 accuracy (‘Acc’). Here we also report average precision (AP), recall (R), precision (P) and F1 score, thus covering both ranking and set-retrieval objectives. AP and R-P curves are obtained by ordering test instances by the raw score given to them by OE, , and rectangle embeddings. Table 1 compares the three systems after using the two sampling protocols to generate folds.
|OE protocol||Sanitized OE protocol|
It is immediately visible that absolute performance numbers are very high under the original OE protocol, for reasons made clear earlier. As soon as the OE protocol is replaced by the sanitized protocol, no system is given any credit for computing transitive closure. The 0/1 accuracy of OE drops from 0.922 to 0.574. F1 score drops even more drastically from 0.916 to 0.262. In contrast, and rectangle embeddings fare better overall, with rectangle embeddings improving beyond .
Whereas and rectangle embeddings improve on OE at the task of set retrieval, their ranking abilities are slightly different. Figure 2 shows that is inferior at ranking to both OE and rectangle embeddings. Rectangle embeddings have the best precision profile at low recall. Modifying our code to use ranking-oriented loss functions (Cao et al., 2007) may address ranking applications better.
7. Concluding remarks
Here we have addressed the problem of completing and relations starting from an incomplete KG, but without corpus support. For out-of-vocabulary (not seen during training) entities, mention contexts in a corpus are vital typing clues (Ling and Weld, 2012; Yaghoobzadeh and Schütze, 2015; Shimaoka et al., 2016). We plan to integrate context (word) embeddings with order and rectangle embeddings. It would be of interest to see how our refined loss objectives and testing protocols compare with other corpus-based methods (Chang et al., 2017; Yamane et al., 2016).
Thanks to Aditya Kusupati and Anand Dhoot for helpful discussions, and nVidia for a GPU grant.
- Athiwaratkun and Wilson (2018) Ben Athiwaratkun and Andrew Gordon Wilson. 2018. On Modeling Hierarchical Data via Probabilistic Order Embeddings. In ICLR. https://openreview.net/forum?id=HJCXZQbAZ
- Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In NIPS Conference. 2787–2795. http://papers.nips.cc/paper/5071-translating-embeddings-for-modeling-multi-relational-data.pdf
- Cao et al. (2007) Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. 2007. Learning to Rank: From Pairwise Approach to Listwise Approach. In ICML. 129–136. http://www.machinelearning.org/proceedings/icml2007/papers/139.pdf
- Chang et al. (2017) Haw-Shiuan Chang, ZiYun Wang, Luke Vilnis, and Andrew McCallum. 2017. Unsupervised Hypernym Detection by Distributional Inclusion Vector Embedding. arXiv preprint arXiv:1710.00880 (2017). https://arxiv.org/pdf/1710.00880.pdf
- Jameel and Schockaert (2016) Shoaib Jameel and Steven Schockaert. 2016. Entity embeddings with conceptual subspaces as a basis for plausible reasoning. arXiv preprint arXiv:1602.05765 (2016).
- Ling and Weld (2012) Xiao Ling and Daniel S Weld. 2012. Fine-Grained Entity Recognition.. In AAAI. http://xiaoling.github.io/pubs/ling-aaai12.pdf
- Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In NIPS Conference. 3111–3119. https://goo.gl/x3DTzS
- Miller et al. (1993) G Miller, R Beckwith, C FellBaum, D Gross, K Miller, and R Tengi. 1993. Five papers on WordNet. Princeton University. ftp://ftp.cogsci.princeton.edu/pub/wordnet/5papers.pdf
- Murty et al. (2017) Shikhar Murty, Patrick Verga, Luke Vilnis, and Andrew McCallum. 2017. Finer Grained Entity Typing with TypeNet. arXiv preprint arXiv:1711.05795 (2017). https://arxiv.org/pdf/1711.05795.pdf
- Nickel (2013) Maximillian Nickel. 2013. Tensor factorization for relational learning. Ph.D. Dissertation. Ludwig–Maximilians–Universität, München. https://edoc.ub.uni-muenchen.de/16056/1/Nickel_Maximilian.pdf
- Nickel and Kiela (2017) Maximillian Nickel and Douwe Kiela. 2017. Poincaré embeddings for learning hierarchical representations. In NIPS Conference. 6341–6350. http://papers.nips.cc/paper/7213-poincare-embeddings-for-learning-hierarchical-representations.pdf
- Nickel et al. (2016) Maximilian Nickel, Lorenzo Rosasco, Tomaso A Poggio, et al. 2016. Holographic Embeddings of Knowledge Graphs. In AAAI Conference. 1955–1961. https://arxiv.org/abs/1510.04935
- Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: Global Vectors for Word Representation.. In EMNLP Conference, Vol. 14. 1532–1543. http://www.emnlp2014.org/papers/pdf/EMNLP2014162.pdf
- Shimaoka et al. (2016) Sonse Shimaoka, Pontus Stenetorp, Kentaro Inui, and Sebastian Riedel. 2016. An Attentive Neural Architecture for Fine-grained Entity Type Classification. arXiv preprint arXiv:1604.05525 (2016). https://arxiv.org/pdf/1604.05525.pdf
- Suchanek et al. (2007) Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007. YAGO: A Core of Semantic Knowledge Unifying WordNet and Wikipedia. In WWW Conference. ACM Press, 697–706. http://www2007.org/papers/paper391.pdf
- Trouillon et al. (2016) Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In ICML. 2071–2080. http://arxiv.org/abs/1606.06357
- Vendrov et al. (2015) Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. 2015. Order-embeddings of images and language. arXiv preprint arXiv:1511.06361 (2015). https://arxiv.org/pdf/1511.06361
- Vilnis and McCallum (2014) Luke Vilnis and Andrew McCallum. 2014. Word representations via gaussian embedding. arXiv preprint arXiv:1412.6623 (2014).
- Xie et al. (2017) Qizhe Xie, Xuezhe Ma, Zihang Dai, and Eduard Hovy. 2017. An Interpretable Knowledge Transfer Model for Knowledge Base Completion. arXiv preprint arXiv:1704.05908 (2017). https://arxiv.org/pdf/1704.05908.pdf
- Yaghoobzadeh and Schütze (2015) Yadollah Yaghoobzadeh and Hinrich Schütze. 2015. Corpus-level Fine-grained Entity Typing Using Contextual Information. In EMNLP Conference. 715–725. http://aclweb.org/anthology/D15-1083
- Yamada et al. (2017) Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. 2017. Learning Distributed Representations of Texts and Entities from Knowledge Base. arXiv preprint arXiv:1705.02494 (2017). https://github.com/studio-ousia/ntee
- Yamane et al. (2016) Josuke Yamane, Tomoya Takatani, Hitoshi Yamada, Makoto Miwa, and Yutaka Sasaki. 2016. Distributional Hypernym Generation by Jointly Learning Clusters and Projections. In COLING. 1871–1879. http://www.aclweb.org/anthology/C16-1176