MMKGR: Multi-hop Multi-modal Knowledge Graph Reasoning

09/03/2022
by   Shangfei Zheng, et al.
2

Multi-modal knowledge graphs (MKGs) include not only the relation triplets, but also related multi-modal auxiliary data (i.e., texts and images), which enhance the diversity of knowledge. However, the natural incompleteness has significantly hindered the applications of MKGs. To tackle the problem, existing studies employ the embedding-based reasoning models to infer the missing knowledge after fusing the multi-modal features. However, the reasoning performance of these methods is limited due to the following problems: (1) ineffective fusion of multi-modal auxiliary features; (2) lack of complex reasoning ability as well as inability to conduct the multi-hop reasoning which is able to infer more missing knowledge. To overcome these problems, we propose a novel model entitled MMKGR (Multi-hop Multi-modal Knowledge Graph Reasoning). Specifically, the model contains the following two components: (1) a unified gate-attention network which is designed to generate effective multi-modal complementary features through sufficient attention interaction and noise reduction; (2) a complementary feature-aware reinforcement learning method which is proposed to predict missing elements by performing the multi-hop reasoning process, based on the features obtained in component (1). The experimental results demonstrate that MMKGR outperforms the state-of-the-art approaches in the MKG reasoning task.

READ FULL TEXT

page 1

page 2

page 11

research
06/17/2023

Do as I can, not as I get: Topology-aware multi-hop reasoning on multi-modal knowledge graphs

Multi-modal knowledge graph (MKG) includes triplets that consist of enti...
research
04/08/2023

DREAM: Adaptive Reinforcement Learning based on Attention Mechanism for Temporal Knowledge Graph Reasoning

Temporal knowledge graphs (TKGs) model the temporal evolution of events ...
research
08/31/2022

Intelligent Traffic Monitoring with Hybrid AI

Challenges in Intelligent Traffic Monitoring (ITMo) are exacerbated by t...
research
09/13/2022

M^2-3DLaneNet: Multi-Modal 3D Lane Detection

Estimating accurate lane lines in 3D space remains challenging due to th...
research
01/17/2022

SQUIRE: A Sequence-to-sequence Framework for Multi-hop Knowledge Graph Reasoning

Multi-hop knowledge graph (KG) reasoning has been widely studied in rece...
research
02/11/2022

Multi-Modal Knowledge Graph Construction and Application: A Survey

Recent years have witnessed the resurgence of knowledge engineering whic...
research
06/17/2023

Query2GMM: Learning Representation with Gaussian Mixture Model for Reasoning over Knowledge Graphs

Logical query answering over Knowledge Graphs (KGs) is a fundamental yet...

Please sign up or login with your details

Forgot password? Click here to reset