Using Graph Algorithms to Pretrain Graph Completion Transformers

10/14/2022
by   Jonathan Pilault, et al.
0

Recent work on Graph Neural Networks has demonstrated that self-supervised pretraining can further enhance performance on downstream graph, link, and node classification tasks. However, the efficacy of pretraining tasks has not been fully investigated for downstream large knowledge graph completion tasks. Using a contextualized knowledge graph embedding approach, we investigate five different pretraining signals, constructed using several graph algorithms and no external data, as well as their combination. We leverage the versatility of our Transformer-based model to explore graph structure generation pretraining tasks, typically inapplicable to most graph embedding methods. We further propose a new path-finding algorithm guided by information gain and find that it is the best-performing pretraining task across three downstream knowledge graph completion datasets. In a multitask setting that combines all pretraining tasks, our method surpasses some of the latest and strong performing knowledge graph embedding methods on all metrics for FB15K-237, on MRR and Hit@1 for WN18RR and on MRR and hit@10 for JF17K (a knowledge hypergraph dataset).

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/17/2022

Deep Bidirectional Language-Knowledge Graph Pretraining

Pretraining a language model (LM) on text has been shown to help various...
research
09/07/2023

Extending Transductive Knowledge Graph Embedding Models for Inductive Logical Relational Inference

Many downstream inference tasks for knowledge graphs, such as relation p...
research
05/26/2023

Commonsense Knowledge Graph Completion Via Contrastive Pretraining and Node Clustering

The nodes in the commonsense knowledge graph (CSKG) are normally represe...
research
06/18/2021

Graph Context Encoder: Graph Feature Inpainting for Graph Generation and Self-supervised Pretraining

We propose the Graph Context Encoder (GCE), a simple but efficient appro...
research
08/21/2016

Probabilistic Knowledge Graph Construction: Compositional and Incremental Approaches

Knowledge graph construction consists of two tasks: extracting informati...
research
03/03/2023

Structure Pretraining and Prompt Tuning for Knowledge Graph Transfer

Knowledge graphs (KG) are essential background knowledge providers in ma...
research
04/16/2020

Hcore-Init: Neural Network Initialization based on Graph Degeneracy

Neural networks are the pinnacle of Artificial Intelligence, as in recen...

Please sign up or login with your details

Forgot password? Click here to reset