State of Compact Architecture Search For Deep Neural Networks

10/15/2019
by   Mohammad Javad Shafiee, et al.
0

The design of compact deep neural networks is a crucial task to enable widespread adoption of deep neural networks in the real-world, particularly for edge and mobile scenarios. Due to the time-consuming and challenging nature of manually designing compact deep neural networks, there has been significant recent research interest into algorithms that automatically search for compact network architectures. A particularly interesting class of compact architecture search algorithms are those that are guided by baseline network architectures. Such algorithms have been shown to be significantly more computationally efficient than unguided methods. In this study, we explore the current state of compact architecture search for deep neural networks through both theoretical and empirical analysis of four different state-of-the-art compact architecture search algorithms: i) group lasso regularization, ii) variational dropout, iii) MorphNet, and iv) Generative Synthesis. We examine these methods in detail based on a number of different factors such as efficiency, effectiveness, and scalability. Furthermore, empirical evaluations are conducted to compare the efficacy of these compact architecture search algorithms across three well-known benchmark datasets. While by no means an exhaustive exploration, we hope that this study helps provide insights into the interesting state of this relatively new area of research in terms of diversity and real, tangible gains already achieved in architecture design improvements. Furthermore, the hope is that this study would help in pushing the conversation forward towards a deeper theoretical and empirical understanding where the research community currently stands in the landscape of compact architecture search for deep neural networks, and the practical challenges and considerations in leveraging such approaches for operational usage.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/11/2022

Automated Architecture Search for Brain-inspired Hyperdimensional Computing

This paper represents the first effort to explore an automated architect...
research
03/26/2019

Deep Demosaicing for Edge Implementation

Most digital cameras use sensors coated with a Color Filter Array (CFA) ...
research
12/07/2018

ShuffleNASNets: Efficient CNN models through modified Efficient Neural Architecture Search

Neural network architectures found by sophistic search algorithms achiev...
research
09/17/2018

FermiNets: Learning generative machines to generate efficient neural networks via generative synthesis

The tremendous potential exhibited by deep learning is often offset by a...
research
03/31/2022

Neural Architecture Search for Speech Emotion Recognition

Deep neural networks have brought significant advancements to speech emo...
research
01/16/2021

Evaluating Online and Offline Accuracy Traversal Algorithms for k-Complete Neural Network Architectures

Architecture sizes for neural networks have been studied widely and seve...
research
06/22/2018

Compact Deep Neural Networks for Computationally Efficient Gesture Classification From Electromyography Signals

Machine learning classifiers using surface electromyography are importan...

Please sign up or login with your details

Forgot password? Click here to reset