DeepAI
Log In Sign Up

Train and You'll Miss It: Interactive Model Iteration with Weak Supervision and Pre-Trained Embeddings

06/26/2020
by   Mayee F. Chen, et al.
1

Our goal is to enable machine learning systems to be trained interactively. This requires models that perform well and train quickly, without large amounts of hand-labeled data. We take a step forward in this direction by borrowing from weak supervision (WS), wherein models can be trained with noisy sources of signal instead of hand-labeled data. But WS relies on training downstream deep networks to extrapolate to unseen data points, which can take hours or days. Pre-trained embeddings can remove this requirement. We do not use the embeddings as features as in transfer learning (TL), which requires fine-tuning for high performance, but instead use them to define a distance function on the data and extend WS source votes to nearby points. Theoretically, we provide a series of results studying how performance scales with changes in source coverage, source accuracy, and the Lipschitzness of label distributions in the embedding space, and compare this rate to standard WS without extension and TL without fine-tuning. On six benchmark NLP and video tasks, our method outperforms WS without extension by 4.1 points, TL without fine-tuning by 12.8 points, and traditionally-supervised deep networks by 13.1 points, and comes within 0.7 points of state-of-the-art weakly-supervised deep networks-all while training in less than half a second.

READ FULL TEXT

page 5

page 6

page 11

page 13

page 14

page 15

page 38

page 40

03/24/2022

Shoring Up the Foundations: Fusing Model Embeddings and Weak Supervision

Foundation models offer an exciting new paradigm for constructing models...
11/08/2021

Improved Regularization and Robustness for Fine-tuning in Neural Networks

A widely used algorithm for transfer learning is fine-tuning, where a pr...
11/06/2016

Beyond Fine Tuning: A Modular Approach to Learning on Small Data

In this paper we present a technique to train neural network models on s...
02/25/2021

Self-Tuning for Data-Efficient Deep Learning

Deep learning has made revolutionary advances to diverse applications in...
12/01/2022

AUG-FedPrompt: Practical Few-shot Federated NLP with Data-augmented Prompts

Transformer-based pre-trained models have become the de-facto solution f...
04/01/2019

Transfer Learning for Clinical Time Series Analysis using Deep Neural Networks

Deep neural networks have shown promising results for various clinical p...
10/04/2018

The Dynamics of Differential Learning I: Information-Dynamics and Task Reachability

We study the topology of the space of learning tasks, which is critical ...

Code Repositories

epoxy

Interactive Model Iteration with Weak Supervision and Pre-Trained Embeddings


view repo