Log In Sign Up

Mask and Reason: Pre-Training Knowledge Graph Transformers for Complex Logical Queries

by   Xiao Liu, et al.

Knowledge graph (KG) embeddings have been a mainstream approach for reasoning over incomplete KGs. However, limited by their inherently shallow and static architectures, they can hardly deal with the rising focus on complex logical queries, which comprise logical operators, imputed edges, multiple source entities, and unknown intermediate entities. In this work, we present the Knowledge Graph Transformer (kgTransformer) with masked pre-training and fine-tuning strategies. We design a KG triple transformation method to enable Transformer to handle KGs, which is further strengthened by the Mixture-of-Experts (MoE) sparse activation. We then formulate the complex logical queries as masked prediction and introduce a two-stage masked pre-training strategy to improve transferability and generalizability. Extensive experiments on two benchmarks demonstrate that kgTransformer can consistently outperform both KG embedding-based baselines and advanced encoders on nine in-domain and out-of-domain reasoning tasks. Additionally, kgTransformer can reason with explainability via providing the full reasoning paths to interpret given answers.


page 1

page 2

page 3

page 4


Query2Particles: Knowledge Graph Reasoning with Particle Embeddings

Answering complex logical queries on incomplete knowledge graphs (KGs) w...

Beta Embeddings for Multi-Hop Logical Reasoning in Knowledge Graphs

One of the fundamental problems in Artificial Intelligence is to perform...

Neural-Symbolic Models for Logical Queries on Knowledge Graphs

Answering complex first-order logic (FOL) queries on knowledge graphs is...

FLEX: Feature-Logic Embedding Framework for CompleX Knowledge Graph Reasoning

Current best performing models for knowledge graph reasoning (KGR) are b...

Unifying Structure Reasoning and Language Model Pre-training for Complex Reasoning

Recent knowledge enhanced pre-trained language models have shown remarka...

A Knowledge Graph for Assessing Aggressive Tax Planning Strategies

The taxation of multi-national companies is a complex field, since it is...

Generating Logical Forms from Graph Representations of Text and Entities

Structured information about entities is critical for many semantic pars...