Elimination of All Bad Local Minima in Deep Learning

01/02/2019
by   Kenji Kawaguchi, et al.
0

In this paper, we theoretically prove that we can eliminate all suboptimal local minima by adding one neuron per output unit to any deep neural network, for multi-class classification, binary classification, and regression with an arbitrary loss function. At every local minimum of any deep neural network with added neurons, the set of parameters of the original neural network (without added neurons) is guaranteed to be a global minimum of the original neural network. The effects of the added neurons are proven to automatically vanish at every local minimum. Unlike many related results in the literature, our theoretical results are directly applicable to common deep learning tasks because the results only rely on the assumptions that automatically hold in the common tasks. Moreover, we discuss several limitations in eliminating the suboptimal local minima in this manner by providing additional theoretical results and several examples.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/22/2018

Adding One Neuron Can Eliminate All Bad Local Minima

One of the main difficulties in analyzing neural networks is the non-con...
research
01/12/2019

Eliminating all bad Local Minima from Loss Landscapes without even adding an Extra Unit

Recent work has noted that all bad local minima can be removed from neur...
research
12/16/2018

Non-attracting Regions of Local Minima in Deep and Wide Neural Networks

Understanding the loss surface of neural networks is essential for the d...
research
11/11/2021

Towards Theoretical Understanding of Flexible Transmitter Networks via Approximation and Local Minima

Flexible Transmitter Network (FTNet) is a recently proposed bio-plausibl...
research
04/07/2019

Every Local Minimum is a Global Minimum of an Induced Model

For non-convex optimization in machine learning, this paper proves that ...
research
05/24/2023

On progressive sharpening, flat minima and generalisation

We present a new approach to understanding the relationship between loss...
research
11/13/2019

Existence of local minima of a minimal 2D pose-graph SLAM problem

In this paper, we show that for a minimal pose-graph problem, even in th...

Please sign up or login with your details

Forgot password? Click here to reset