NER-BERT: A Pre-trained Model for Low-Resource Entity Tagging

by   Zihan Liu, et al.

Named entity recognition (NER) models generally perform poorly when large training datasets are unavailable for low-resource domains. Recently, pre-training a large-scale language model has become a promising direction for coping with the data scarcity issue. However, the underlying discrepancies between the language modeling and NER task could limit the models' performance, and pre-training for the NER task has rarely been studied since the collected NER datasets are generally small or large but with low quality. In this paper, we construct a massive NER corpus with a relatively high quality, and we pre-train a NER-BERT model based on the created dataset. Experimental results show that our pre-trained model can significantly outperform BERT as well as other strong baselines in low-resource scenarios across nine diverse domains. Moreover, a visualization of entity representations further indicates the effectiveness of NER-BERT for categorizing a variety of entities.



There are no comments yet.


page 4

page 15


Coarse-to-Fine Pre-training for Named Entity Recognition

More recently, Named Entity Recognition hasachieved great advances aided...

LightNER: A Lightweight Generative Framework with Prompt-guided Attention for Low-resource NER

Most existing NER methods rely on extensive labeled data for model train...

newsSweeper at SemEval-2020 Task 11: Context-Aware Rich Feature Representations For Propaganda Classification

This paper describes our submissions to SemEval 2020 Task 11: Detection ...

Improving NER's Performance with Massive financial corpus

Training large deep neural networks needs massive high quality annotatio...

One Model to Recognize Them All: Marginal Distillation from NER Models with Different Tag Sets

Named entity recognition (NER) is a fundamental component in the modern ...

Green CWS: Extreme Distillation and Efficient Decode Method Towards Industrial Application

Benefiting from the strong ability of the pre-trained model, the researc...

Low-Resource Neural Headline Generation

Recent neural headline generation models have shown great results, but a...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.