NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm

11/06/2017
by   Xiaoliang Dai, et al.
0

Neural networks (NNs) have begun to have a pervasive impact on various applications of machine learning. However, the problem of finding an optimal NN architecture for large applications has remained open for several decades. Conventional approaches search for the optimal NN architecture through extensive trial-and-error. Such a procedure is quite inefficient. In addition, the generated NN architectures incur substantial redundancy. To address these problems, we propose an NN synthesis tool (NeST) that automatically generates very compact architectures for a given dataset. NeST starts with a seed NN architecture. It iteratively tunes the architecture with gradient-based growth and magnitude-based pruning of neurons and connections. Our experimental results show that NeST yields accurate yet very compact NNs with a wide range of seed architecture selection. For example, for the LeNet-300-100 (LeNet-5) NN architecture derived from the MNIST dataset, we reduce network parameters by 34.1x (74.3x) and floating-point operations (FLOPs) by 35.8x (43.7x). For the AlexNet NN architecture derived from the ImageNet dataset, we reduce network parameters by 15.7x and FLOPs by 4.6x. All these results are the current state-of-the-art for these architectures.

READ FULL TEXT
research
12/12/2019

STEERAGE: Synthesis of Neural Networks Using Architecture Search and Grow-and-Prune Methods

Neural networks (NNs) have been successfully deployed in many applicatio...
research
05/27/2019

Incremental Learning Using a Grow-and-Prune Paradigm with Efficient Neural Networks

Deep neural networks (DNNs) have become a widely deployed model for nume...
research
05/27/2019

CGaP: Continuous Growth and Pruning for Efficient Deep Learning

Today a canonical approach to reduce the computation cost of Deep Neural...
research
04/20/2020

Two-Level Lattice Neural Network Architectures for Control of Nonlinear Systems

In this paper, we consider the problem of automatically designing a Rect...
research
05/30/2018

Grow and Prune Compact, Fast, and AccurateLSTMs

Long short-term memory (LSTM) has been widely used for sequential data m...
research
06/05/2020

Neural Network Calculator for Designing Trojan Detectors

This work presents a web-based interactive neural network (NN) calculato...
research
02/25/2019

Modularity as a Means for Complexity Management in Neural Networks Learning

Training a Neural Network (NN) with lots of parameters or intricate arch...

Please sign up or login with your details

Forgot password? Click here to reset