Log In Sign Up

Jointly Improving Language Understanding and Generation with Quality-Weighted Weak Supervision of Automatic Labeling

by   Ernie Chang, et al.

Neural natural language generation (NLG) and understanding (NLU) models are data-hungry and require massive amounts of annotated data to be competitive. Recent frameworks address this bottleneck with generative models that synthesize weak labels at scale, where a small amount of training labels are expert-curated and the rest of the data is automatically annotated. We follow that approach, by automatically constructing a large-scale weakly-labeled data with a fine-tuned GPT-2, and employ a semi-supervised framework to jointly train the NLG and NLU models. The proposed framework adapts the parameter updates to the models according to the estimated label-quality. On both the E2E and Weather benchmarks, we show that this weakly supervised training paradigm is an effective approach under low resource scenarios and outperforming benchmark systems on both datasets when 100


page 1

page 2

page 3

page 4


WALNUT: A Benchmark on Weakly Supervised Learning for Natural Language Understanding

Building quality machine learning models for natural language understand...

Fidelity-Weighted Learning

Training deep neural networks requires many training samples, but in pra...

Learning Fast Matching Models from Weak Annotations

This paper proposes a novel training scheme for fast matching models in ...

Learning the Structure of Generative Models without Labeled Data

Curating labeled training data has become the primary bottleneck in mach...

Avoiding Your Teacher's Mistakes: Training Neural Networks with Controlled Weak Supervision

Training deep neural networks requires massive amounts of training data,...

Learning to Learn from Weak Supervision by Full Supervision

In this paper, we propose a method for training neural networks when we ...