NER-BERT: A Pre-trained Model for Low-Resource Entity Tagging

12/01/2021
by   Zihan Liu, et al.
11

Named entity recognition (NER) models generally perform poorly when large training datasets are unavailable for low-resource domains. Recently, pre-training a large-scale language model has become a promising direction for coping with the data scarcity issue. However, the underlying discrepancies between the language modeling and NER task could limit the models' performance, and pre-training for the NER task has rarely been studied since the collected NER datasets are generally small or large but with low quality. In this paper, we construct a massive NER corpus with a relatively high quality, and we pre-train a NER-BERT model based on the created dataset. Experimental results show that our pre-trained model can significantly outperform BERT as well as other strong baselines in low-resource scenarios across nine diverse domains. Moreover, a visualization of entity representations further indicates the effectiveness of NER-BERT for categorizing a variety of entities.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 4

page 15

10/16/2020

Coarse-to-Fine Pre-training for Named Entity Recognition

More recently, Named Entity Recognition hasachieved great advances aided...
08/31/2021

LightNER: A Lightweight Generative Framework with Prompt-guided Attention for Low-resource NER

Most existing NER methods rely on extensive labeled data for model train...
07/21/2020

newsSweeper at SemEval-2020 Task 11: Context-Aware Rich Feature Representations For Propaganda Classification

This paper describes our submissions to SemEval 2020 Task 11: Detection ...
07/31/2020

Improving NER's Performance with Massive financial corpus

Training large deep neural networks needs massive high quality annotatio...
04/10/2020

One Model to Recognize Them All: Marginal Distillation from NER Models with Different Tag Sets

Named entity recognition (NER) is a fundamental component in the modern ...
11/17/2021

Green CWS: Extreme Distillation and Efficient Decode Method Towards Industrial Application

Benefiting from the strong ability of the pre-trained model, the researc...
07/31/2017

Low-Resource Neural Headline Generation

Recent neural headline generation models have shown great results, but a...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.