A Simple but Effective Pluggable Entity Lookup Table for Pre-trained Language Models

02/27/2022
by   Deming Ye, et al.
0

Pre-trained language models (PLMs) cannot well recall rich factual knowledge of entities exhibited in large-scale corpora, especially those rare entities. In this paper, we propose to build a simple but effective Pluggable Entity Lookup Table (PELT) on demand by aggregating the entity's output representations of multiple occurrences in the corpora. PELT can be compatibly plugged as inputs to infuse supplemental entity knowledge into PLMs. Compared to previous knowledge-enhanced PLMs, PELT only requires 0.2 with capability of acquiring knowledge from out-of-domain corpora for domain adaptation scenario. The experiments on knowledge-related tasks demonstrate that our method, PELT, can flexibly and effectively transfer entity knowledge from related corpora into PLMs with different architectures.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/03/2023

Table and Image Generation for Investigating Knowledge of Entities in Pre-trained Vision and Language Models

In this paper, we propose a table and image generation task to verify ho...
research
07/30/2021

Towards Continual Entity Learning in Language Models for Conversational Agents

Neural language models (LM) trained on diverse corpora are known to work...
research
08/20/2021

SMedBERT: A Knowledge-Enhanced Pre-trained Language Model with Structured Semantics for Medical Text Mining

Recently, the performance of Pre-trained Language Models (PLMs) has been...
research
02/03/2022

Towards Coherent and Consistent Use of Entities in Narrative Generation

Large pre-trained language models (LMs) have demonstrated impressive cap...
research
10/01/2020

CoLAKE: Contextualized Language and Knowledge Embedding

With the emerging branch of incorporating factual knowledge into pre-tra...
research
10/16/2021

Metadata Shaping: Natural Language Annotations for the Tail

Language models (LMs) have made remarkable progress, but still struggle ...
research
02/16/2021

Boosting Low-Resource Biomedical QA via Entity-Aware Masking Strategies

Biomedical question-answering (QA) has gained increased attention for it...

Please sign up or login with your details

Forgot password? Click here to reset