A Sample-Based Training Method for Distantly Supervised Relation Extraction with Pre-Trained Transformers

by   Mehrdad Nasser, et al.

Multiple instance learning (MIL) has become the standard learning paradigm for distantly supervised relation extraction (DSRE). However, due to relation extraction being performed at bag level, MIL has significant hardware requirements for training when coupled with large sentence encoders such as deep transformer neural networks. In this paper, we propose a novel sampling method for DSRE that relaxes these hardware requirements. In the proposed method, we limit the number of sentences in a batch by randomly sampling sentences from the bags in the batch. However, this comes at the cost of losing valid sentences from bags. To alleviate the issues caused by random sampling, we use an ensemble of trained models for prediction. We demonstrate the effectiveness of our approach by using our proposed learning setting to fine-tuning BERT on the widely NYT dataset. Our approach significantly outperforms previous state-of-the-art methods in terms of AUC and P@N metrics.


page 1

page 2

page 3

page 4


An Empirical Study of Using Pre-trained BERT Models for Vietnamese Relation Extraction Task at VLSP 2020

In this paper, we present an empirical study of using pre-trained BERT m...

Improving Distantly-Supervised Relation Extraction through BERT-based Label Instance Embeddings

Distantly-supervised relation extraction (RE) is an effective method to ...

Fine-tuning Pre-Trained Transformer Language Models to Distantly Supervised Relation Extraction

Distantly supervised relation extraction is widely used to extract relat...

Finding Influential Instances for Distantly Supervised Relation Extraction

Distant supervision has been demonstrated to be highly beneficial to enh...

Ensemble Making Few-Shot Learning Stronger

Few-shot learning has been proposed and rapidly emerging as a viable mea...

Can Fine-tuning Pre-trained Models Lead to Perfect NLP? A Study of the Generalizability of Relation Extraction

Fine-tuning pre-trained models have achieved impressive performance on s...

DNNRE: A Dynamic Neural Network for Distant Supervised Relation Extraction

Distant Supervised Relation Extraction (DSRE) is usually formulated as a...