Mask and Reason: Pre-Training Knowledge Graph Transformers for Complex Logical Queries

08/16/2022
by   Xiao Liu, et al.
0

Knowledge graph (KG) embeddings have been a mainstream approach for reasoning over incomplete KGs. However, limited by their inherently shallow and static architectures, they can hardly deal with the rising focus on complex logical queries, which comprise logical operators, imputed edges, multiple source entities, and unknown intermediate entities. In this work, we present the Knowledge Graph Transformer (kgTransformer) with masked pre-training and fine-tuning strategies. We design a KG triple transformation method to enable Transformer to handle KGs, which is further strengthened by the Mixture-of-Experts (MoE) sparse activation. We then formulate the complex logical queries as masked prediction and introduce a two-stage masked pre-training strategy to improve transferability and generalizability. Extensive experiments on two benchmarks demonstrate that kgTransformer can consistently outperform both KG embedding-based baselines and advanced encoders on nine in-domain and out-of-domain reasoning tasks. Additionally, kgTransformer can reason with explainability via providing the full reasoning paths to interpret given answers.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/22/2020

Beta Embeddings for Multi-Hop Logical Reasoning in Knowledge Graphs

One of the fundamental problems in Artificial Intelligence is to perform...
research
05/16/2022

Neural-Symbolic Models for Logical Queries on Knowledge Graphs

Answering complex first-order logic (FOL) queries on knowledge graphs is...
research
09/07/2023

Extending Transductive Knowledge Graph Embedding Models for Inductive Logical Relational Inference

Many downstream inference tasks for knowledge graphs, such as relation p...
research
06/06/2023

Logic Diffusion for Knowledge Graph Reasoning

Most recent works focus on answering first order logical queries to expl...
research
09/14/2023

A Data Source for Reasoning Embodied Agents

Recent progress in using machine learning models for reasoning tasks has...
research
05/21/2019

Generating Logical Forms from Graph Representations of Text and Entities

Structured information about entities is critical for many semantic pars...
research
08/12/2020

A Knowledge Graph for Assessing Aggressive Tax Planning Strategies

The taxation of multi-national companies is a complex field, since it is...

Please sign up or login with your details

Forgot password? Click here to reset