Evaluating Lottery Tickets Under Distributional Shifts

10/28/2019
by   Shrey Desai, et al.
0

The Lottery Ticket Hypothesis suggests large, over-parameterized neural networks consist of small, sparse subnetworks that can be trained in isolation to reach a similar (or better) test accuracy. However, the initialization and generalizability of the obtained sparse subnetworks have been recently called into question. Our work focuses on evaluating the initialization of sparse subnetworks under distributional shifts. Specifically, we investigate the extent to which a sparse subnetwork obtained in a source domain can be re-trained in isolation in a dissimilar, target domain. In addition, we examine the effects of different initialization strategies at transfer-time. Our experiments show that sparse subnetworks obtained through lottery ticket training do not simply overfit to particular domains, but rather reflect an inductive bias of deep neural networks that can be exploited in multiple domains.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/25/2020

Data Parallelism in Training Sparse Neural Networks

Network pruning is an effective methodology to compress large neural net...
research
02/10/2020

Calibrate and Prune: Improving Reliability of Lottery Tickets Through Prediction Calibration

The hypothesis that sub-network initializations (lottery) exist within t...
research
09/01/2021

Evaluating Predictive Uncertainty under Distributional Shift on Dialogue Dataset

In open-domain dialogues, predictive uncertainties are mainly evaluated ...
research
05/22/2018

Adapted Deep Embeddings: A Synthesis of Methods for k-Shot Inductive Transfer Learning

The focus in machine learning has branched beyond training classifiers o...
research
05/19/2019

Sparse Transfer Learning via Winning Lottery Tickets

The recently proposed Lottery Ticket Hypothesis of Frankle and Carbin (2...
research
12/11/2019

Linear Mode Connectivity and the Lottery Ticket Hypothesis

We introduce "instability analysis," a framework for assessing whether t...
research
08/31/2018

Nightmare at test time: How punctuation prevents parsers from generalizing

Punctuation is a strong indicator of syntactic structure, and parsers tr...

Please sign up or login with your details

Forgot password? Click here to reset