The rise of the lottery heroes: why zero-shot pruning is hard

02/24/2022
by   Enzo Tartaglione, et al.
0

Recent advances in deep learning optimization showed that just a subset of parameters are really necessary to successfully train a model. Potentially, such a discovery has broad impact from the theory to application; however, it is known that finding these trainable sub-network is a typically costly process. This inhibits practical applications: can the learned sub-graph structures in deep learning models be found at training time? In this work we explore such a possibility, observing and motivating why common approaches typically fail in the extreme scenarios of interest, and proposing an approach which potentially enables training with reduced computational effort. The experiments on either challenging architectures and datasets suggest the algorithmic accessibility over such a computational gain, and in particular a trade-off between accuracy achieved and training complexity deployed emerges.

READ FULL TEXT
research
05/17/2022

Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey

State-of-the-art deep learning models have a parameter count that reache...
research
03/06/2020

Towards Practical Lottery Ticket Hypothesis for Adversarial Training

Recent research has proposed the lottery ticket hypothesis, suggesting t...
research
12/02/2022

Legal Prompting: Teaching a Language Model to Think Like a Lawyer

Large language models that are capable of zero or few-shot prompting app...
research
07/19/2022

To update or not to update? Neurons at equilibrium in deep models

Recent advances in deep learning optimization showed that, with some a-p...
research
02/17/2023

A New Baseline for GreenAI: Finding the Optimal Sub-Network via Layer and Channel Pruning

The concept of Green AI has been gaining attention within the deep learn...
research
06/27/2023

CLIPA-v2: Scaling CLIP Training with 81.1 within a $10,000 Budget; An Extra $4,000 Unlocks 81.8

The recent work CLIPA presents an inverse scaling law for CLIP training ...
research
11/27/2022

Neural Font Rendering

Recent advances in deep learning techniques and applications have revolu...

Please sign up or login with your details

Forgot password? Click here to reset