Energy-Aware Neural Architecture Optimization with Fast Splitting Steepest Descent

by   Dilin Wang, et al.

Designing energy-efficient networks is of critical importance for enabling state-of-the-art deep learning in mobile and edge settings where the computation and energy budgets are highly limited. Recently, Wu et al. (2019) framed the search of efficient neural architectures into a continuous splitting process: it iteratively splits existing neurons into multiple off-springs to achieve progressive loss minimization, thus finding novel architectures by gradually growing the neural network. However, this method was not specifically tailored for designing energy-efficient networks, and is computationally expensive on large-scale benchmarks. In this work, we substantially improve Wu et al. (2019) in two significant ways: 1) we incorporate the energy cost of splitting different neurons to better guide the splitting process, thereby discovering more energy-efficient network architectures; 2) we substantially speed up the splitting process of Wu et al. (2019), which requires expensive eigen-decomposition, by proposing a highly scalable Rayleigh-quotient stochastic gradient algorithm. Our fast algorithm allows us to reduce the computational cost of splitting to the same level of typical back-propagation updates and enables efficient implementation on GPU. Extensive empirical results show that our method can train highly accurate and energy-efficient networks on challenging datasets such as ImageNet, improving a variety of baselines, including the pruning-based methods and expert-designed architectures.



There are no comments yet.


page 1

page 2

page 3

page 4


Steepest Descent Neural Architecture Optimization: Escaping Local Optimum with Signed Neural Splitting

We propose signed splitting steepest descent (S3D), which progressively ...

Pruning untrained neural networks: Principles and Analysis

Overparameterized neural networks display state-of-the art performance. ...

Splitting Steepest Descent for Growing Neural Architectures

We develop a progressive training approach for neural networks which ada...

Neural Architecture Search for Energy Efficient Always-on Audio Models

Mobile and edge computing devices for always-on audio classification req...

To prune, or not to prune: exploring the efficacy of pruning for model compression

Model pruning seeks to induce sparsity in a deep neural network's variou...

Efficient Lottery Ticket Finding: Less Data is More

The lottery ticket hypothesis (LTH) reveals the existence of winning tic...

SuperNet – An efficient method of neural networks ensembling

The main flaw of neural network ensembling is that it is exceptionally d...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.