Automatic Rule Induction for Efficient Semi-Supervised Learning

05/18/2022
by   Reid Pryzant, et al.
0

Semi-supervised learning has shown promise in allowing NLP models to generalize from small amounts of labeled data. Meanwhile, pretrained transformer models act as black-box correlation engines that are difficult to explain and sometimes behave unreliably. In this paper, we propose tackling both of these challenges via Automatic Rule Induction (ARI), a simple and general-purpose framework for the automatic discovery and integration of symbolic rules into pretrained transformer models. First, we extract weak symbolic rules from low-capacity machine learning models trained on small amounts of labeled data. Next, we use an attention mechanism to integrate these rules into high-capacity pretrained transformer models. Last, the rule-augmented system becomes part of a self-training framework to boost supervision signal on unlabeled data. These steps can be layered beneath a variety of existing weak supervision and semi-supervised NLP algorithms in order to improve performance and interpretability. Experiments across nine sequence classification and relation extraction tasks suggest that ARI can improve state-of-the-art methods with no manual effort and minimal computational overhead.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/12/2021

Self-Training with Weak Supervision

State-of-the-art deep neural networks require large-scale labeled traini...
research
11/19/2019

End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures

We study ResNet-, Time-Depth Separable ConvNets-, and Transformer-based ...
research
10/19/2021

Neural Medication Extraction: A Comparison of Recent Models in Supervised and Semi-supervised Learning Settings

Drug prescriptions are essential information that must be encoded in ele...
research
09/02/2021

Transfer of Pretrained Model Weights Substantially Improves Semi-Supervised Image Classification

Deep neural networks produce state-of-the-art results when trained on a ...
research
10/09/2020

Denoising Multi-Source Weak Supervision for Neural Text Classification

We study the problem of learning neural text classifiers without using a...
research
02/18/2016

Boost Picking: A Universal Method on Converting Supervised Classification to Semi-supervised Classification

This paper proposes a universal method, Boost Picking, to train supervis...
research
07/10/2023

Learning to Solve Constraint Satisfaction Problems with Recurrent Transformer

Constraint satisfaction problems (CSPs) are about finding values of vari...

Please sign up or login with your details

Forgot password? Click here to reset