Learning across scales - A multiscale method for Convolution Neural Networks

by   Eldad Haber, et al.
Emory University
Xtract AI
The University of British Columbia

In this work we establish the relation between optimal control and training deep Convolution Neural Networks (CNNs). We show that the forward propagation in CNNs can be interpreted as a time-dependent nonlinear differential equation and learning as controlling the parameters of the differential equation such that the network approximates the data-label relation for given training data. Using this continuous interpretation we derive two new methods to scale CNNs with respect to two different dimensions. The first class of multiscale methods connects low-resolution and high-resolution data through prolongation and restriction of CNN parameters. We demonstrate that this enables classifying high-resolution images using CNNs trained with low-resolution images and vice versa and warm-starting the learning process. The second class of multiscale methods connects shallow and deep networks and leads to new training strategies that gradually increase the depths of the CNN while re-using parameters for initializations.


On Underdamped Nesterov's Acceleration

The high-resolution differential equation framework has been proven to b...

Layer-Parallel Training of Deep Residual Neural Networks

Residual neural networks (ResNets) are a promising class of deep neural ...

Multigrid-Augmented Deep Learning for the Helmholtz Equation: Better Scalability with Compact Implicit Layers

We present a deep learning-based iterative approach to solve the discret...

Super-resolving Dark Matter Halos using Generative Deep Learning

Generative deep learning methods built upon Convolutional Neural Network...

Multi-Grid Back-Projection Networks

Multi-Grid Back-Projection (MGBP) is a fully-convolutional network archi...

WPPNets: Unsupervised CNN Training with Wasserstein Patch Priors for Image Superresolution

We introduce WPPNets, which are CNNs trained by a new unsupervised loss ...

High-Resolution Convolutional Neural Networks on Homomorphically Encrypted Data via Sharding Ciphertexts

Recently, Deep Convolutional Neural Networks (DCNNs) including the ResNe...

Please sign up or login with your details

Forgot password? Click here to reset