DeepAI AI Chat
Log In Sign Up

Multi-modal Entity Alignment in Hyperbolic Space

by   Hao Guo, et al.

Many AI-related tasks involve the interactions of data in multiple modalities. It has been a new trend to merge multi-modal information into knowledge graph(KG), resulting in multi-modal knowledge graphs (MMKG). However, MMKGs usually suffer from low coverage and incompleteness. To mitigate this problem, a viable approach is to integrate complementary knowledge from other MMKGs. To this end, although existing entity alignment approaches could be adopted, they operate in the Euclidean space, and the resulting Euclidean entity representations can lead to large distortion of KG's hierarchical structure. Besides, the visual information has yet not been well exploited. In response to these issues, in this work, we propose a novel multi-modal entity alignment approach, Hyperbolic multi-modal entity alignment(HMEA), which extends the Euclidean representation to hyperboloid manifold. We first adopt the Hyperbolic Graph Convolutional Networks (HGCNs) to learn structural representations of entities. Regarding the visual information, we generate image embeddings using the densenet model, which are also projected into the hyperbolic space using HGCNs. Finally, we combine the structure and visual representations in the hyperbolic space and use the aggregated embeddings to predict potential alignment results. Extensive experiments and ablation studies demonstrate the effectiveness of our proposed model and its components.


page 2

page 6

page 8


Vision, Deduction and Alignment: An Empirical Study on Multi-modal Knowledge Graph Alignment

Entity alignment (EA) for knowledge graphs (KGs) plays a critical role i...

Attribute-Consistent Knowledge Graph Representation Learning for Multi-Modal Entity Alignment

The multi-modal entity alignment (MMEA) aims to find all equivalent enti...

Multi-modal Contrastive Representation Learning for Entity Alignment

Multi-modal entity alignment aims to identify equivalent entities betwee...

MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybrid

As an important variant of entity alignment (EA), multi-modal entity ali...

Boosting Entity-aware Image Captioning with Multi-modal Knowledge Graph

Entity-aware image captioning aims to describe named entities and events...

A Fully Hyperbolic Neural Model for Hierarchical Multi-Class Classification

Label inventories for fine-grained entity typing have grown in size and ...

Relational Graph Learning on Visual and Kinematics Embeddings for Accurate Gesture Recognition in Robotic Surgery

Automatic surgical gesture recognition is fundamentally important to ena...