Embedding Multimodal Relational Data for Knowledge Base Completion

by   Pouya Pezeshkpour, et al.

Representing entities and relations in an embedding space is a well-studied approach for machine learning on relational data. Existing approaches, however, primarily focus on simple link structure between a finite set of entities, ignoring the variety of data types that are often used in knowledge bases, such as text, images, and numerical values. In this paper, we propose multimodal knowledge base embeddings (MKBE) that use different neural encoders for this variety of observed data, and combine them with existing relational models to learn embeddings of the entities and multimodal data. Further, using these learned embedings and different neural decoders, we introduce a novel multimodal imputation model to generate missing multimodal values, like text and images, from information in the knowledge base. We enrich existing relational datasets to create two novel benchmarks that contain additional information such as textual descriptions and images of the original entities. We demonstrate that our models utilize this additional information effectively to provide more accurate link prediction, achieving state-of-the-art results with a considerable gap of 5-7 quality of our generated multimodal values via a user study. We have release the datasets and the open-source implementation of our models at https://github.com/pouyapez/mkbe.


HyperKG: Hyperbolic Knowledge Graph Embeddings for Knowledge Base Completion

Learning embeddings of entities and relations existing in knowledge base...

Stochastic Neighbor Embedding of Multimodal Relational Data for Image-Text Simultaneous Visualization

Multimodal relational data analysis has become of increasing importance ...

Predicting affinity ties in a surname network

From administrative registers of last names in Santiago, Chile, we creat...

Learning New Facts From Knowledge Bases With Neural Tensor Networks and Semantic Word Vectors

Knowledge bases provide applications with the benefit of easily accessib...

Combining Two And Three-Way Embeddings Models for Link Prediction in Knowledge Bases

This paper tackles the problem of endogenous link prediction for Knowled...

A Systematic Investigation of KB-Text Embedding Alignment at Scale

Knowledge bases (KBs) and text often contain complementary knowledge: KB...

Investigating Robustness and Interpretability of Link Prediction via Adversarial Modifications

Representing entities and relations in an embedding space is a well-stud...

Please sign up or login with your details

Forgot password? Click here to reset