Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning

04/29/2020
by   Tao Shen, et al.
0

In this work, we aim at equipping pre-trained language models with structured knowledge. We present two self-supervised tasks learning over raw text with the guidance from knowledge graphs. Building upon entity-level masked language models, our first contribution is an entity masking scheme that exploits relational knowledge underlying the text. This is fulfilled by using a linked knowledge graph to select informative entities and then masking their mentions. In addition we use knowledge graphs to obtain distractors for the masked entities, and propose a novel distractor-suppressed ranking objective which is optimized jointly with masked language model. In contrast to existing paradigms, our approach uses knowledge graphs implicitly, only during pre-training, to inject language models with structured knowledge via learning from raw text. It is more efficient than retrieval-based methods that perform entity linking and integration during finetuning and inference, and generalizes more effectively than the methods that directly learn from concatenated graph triples. Experiments show that our proposed model achieves improved performance on five benchmark datasets, including question answering and knowledge base completion tasks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/04/2022

KGLM: Integrating Knowledge Graph Structure in Language Models for Link Prediction

The ability of knowledge graphs to represent complex relationships at sc...
research
05/17/2022

SKILL: Structured Knowledge Infusion for Large Language Models

Large language models (LLMs) have demonstrated human-level performance o...
research
04/28/2022

Instilling Type Knowledge in Language Models via Multi-Task QA

Understanding human language often necessitates understanding entities a...
research
08/20/2021

SMedBERT: A Knowledge-Enhanced Pre-trained Language Model with Structured Semantics for Medical Text Mining

Recently, the performance of Pre-trained Language Models (PLMs) has been...
research
10/25/2022

KnowGL: Knowledge Generation and Linking from Text

We propose KnowGL, a tool that allows converting text into structured re...
research
05/28/2023

MemeGraphs: Linking Memes to Knowledge Graphs

Memes are a popular form of communicating trends and ideas in social med...
research
01/27/2022

Ontology-enhanced Prompt-tuning for Few-shot Learning

Few-shot Learning (FSL) is aimed to make predictions based on a limited ...

Please sign up or login with your details

Forgot password? Click here to reset