Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor Methods

06/28/2015
by   Majid Janzamin, et al.
0

Training neural networks is a challenging non-convex optimization problem, and backpropagation or gradient descent can get stuck in spurious local optima. We propose a novel algorithm based on tensor decomposition for guaranteed training of two-layer neural networks. We provide risk bounds for our proposed method, with a polynomial sample complexity in the relevant parameters, such as input dimension and number of neurons. While learning arbitrary target functions is NP-hard, we provide transparent conditions on the function and the input for learnability. Our training method is based on tensor decomposition, which provably converges to the global optimum, under a set of mild non-degeneracy conditions. It consists of simple embarrassingly parallel linear and multi-linear operations, and is competitive with standard stochastic gradient descent (SGD), in terms of computational complexity. Thus, we propose a computationally efficient method with guaranteed risk bounds for training neural networks with one hidden layer.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

12/08/2014

Provable Methods for Training Neural Networks with Sparse Connectivity

We provide novel guaranteed approaches for training feedforward neural n...
02/01/2019

DANTE: Deep AlterNations for Training nEural networks

We present DANTE, a novel method for training neural networks using the ...
02/17/2022

Training neural networks using monotone variational inequality

Despite the vast empirical success of neural networks, theoretical under...
10/11/2015

Neural Networks with Few Multiplications

For most deep learning algorithms training is notoriously time consuming...
10/22/2020

Beyond Lazy Training for Over-parameterized Tensor Decomposition

Over-parametrization is an important technique in training neural networ...
01/13/2020

Backward Feature Correction: How Deep Learning Performs Deep Learning

How does a 110-layer ResNet learn a high-complexity classifier using rel...
09/07/2019

Towards Understanding the Importance of Noise in Training Neural Networks

Numerous empirical evidence has corroborated that the noise plays a cruc...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.