Splitting Steepest Descent for Growing Neural Architectures

10/06/2019
by   Qiang Liu, et al.
0

We develop a progressive training approach for neural networks which adaptively grows the network structure by splitting existing neurons to multiple off-springs. By leveraging a functional steepest descent idea, we derive a simple criterion for deciding the best subset of neurons to split and a splitting gradient for optimally updating the off-springs. Theoretically, our splitting strategy is a second-order functional steepest descent for escaping saddle points in an ∞-Wasserstein metric space, on which the standard parametric gradient descent is a first-order steepest descent. Our method provides a new computationally efficient approach for optimizing neural network structures, especially for learning lightweight neural architectures in resource-constrained settings.

READ FULL TEXT

page 7

page 19

research
03/23/2020

Steepest Descent Neural Architecture Optimization: Escaping Local Optimum with Signed Neural Splitting

We propose signed splitting steepest descent (S3D), which progressively ...
research
10/07/2019

Energy-Aware Neural Architecture Optimization with Fast Splitting Steepest Descent

Designing energy-efficient networks is of critical importance for enabli...
research
02/09/2020

Splitting Convolutional Neural Network Structures for Efficient Inference

For convolutional neural networks (CNNs) that have a large volume of inp...
research
04/19/2020

Stochastic gradient algorithms from ODE splitting perspective

We present a different view on stochastic optimization, which goes back ...
research
12/12/2019

Adaptive Reticulum

Neural Networks and Random Forests: two popular techniques for supervise...
research
09/09/2020

Tunable Subnetwork Splitting for Model-parallelism of Neural Network Training

Alternating minimization methods have recently been proposed as alternat...
research
09/23/2022

I-SPLIT: Deep Network Interpretability for Split Computing

This work makes a substantial step in the field of split computing, i.e....

Please sign up or login with your details

Forgot password? Click here to reset