Efficient Architecture Search by Network Transformation

07/16/2017
by   Han Cai, et al.
0

Techniques for automatically designing deep neural network architectures such as reinforcement learning based approaches have recently shown promising results. However, their success is based on vast computational resources (e.g. hundreds of GPUs), making them difficult to be widely used. A noticeable limitation is that they still design and train each network from scratch during the exploration of the architecture space, which is highly inefficient. In this paper, we propose a new framework toward efficient architecture search by exploring the architecture space based on the current network and reusing its weights. We employ a reinforcement learning agent as the meta-controller, whose action is to grow the network depth or layer width with function-preserving transformations. As such, the previously validated networks can be reused for further exploration, thus saves a large amount of computational cost. We apply our method to explore the architecture space of the plain convolutional neural networks (no skip-connections, branching etc.) on image benchmark datasets (CIFAR-10, SVHN) with restricted computational resources (5 GPUs). Our method can design highly competitive networks that outperform existing networks using the same design scheme. On CIFAR-10, our model without skip-connections achieves 4.23% test error rate, exceeding a vast majority of modern architectures and approaching DenseNet. Furthermore, by applying our method to explore the DenseNet architecture space, we are able to achieve more accurate networks with fewer parameters.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/07/2018

Path-Level Network Transformation for Efficient Architecture Search

We introduce a new function-preserving transformation for efficient neur...
research
11/13/2017

Simple And Efficient Architecture Search for Convolutional Neural Networks

Neural networks have recently had a lot of success for many tasks. Howev...
research
09/04/2019

Rethinking the Number of Channels for the Convolutional Neural Network

Latest algorithms for automatic neural architecture search perform remar...
research
12/12/2019

STEERAGE: Synthesis of Neural Networks Using Architecture Search and Grow-and-Prune Methods

Neural networks (NNs) have been successfully deployed in many applicatio...
research
05/06/2019

Fast and Reliable Architecture Selection for Convolutional Neural Networks

The performance of a Convolutional Neural Network (CNN) depends on its h...
research
01/30/2022

Augmenting Novelty Search with a Surrogate Model to Engineer Meta-Diversity in Ensembles of Classifiers

Using Neuroevolution combined with Novelty Search to promote behavioural...
research
12/03/2019

Multi-Criterion Evolutionary Design of Deep Convolutional Neural Networks

Convolutional neural networks (CNNs) are the backbones of deep learning ...

Please sign up or login with your details

Forgot password? Click here to reset