Deep linear neural networks with arbitrary loss: All local minima are global

12/05/2017
by   Thomas Laurent, et al.
0

We consider deep linear networks with arbitrary differentiable loss. We provide a short and elementary proof of the following fact: all local minima are global minima if each hidden layer is wider than either the input or output layer.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/28/2019

Depth creates no more spurious local minima

We show that for any convex differentiable loss function, a deep linear ...
research
02/16/2019

Making Convex Loss Functions Robust to Outliers using e-Exponentiated Transformation

In this paper, we propose a novel e-exponentiated transformation, 0.5< e...
research
02/11/2020

Unique Properties of Wide Minima in Deep Networks

It is well known that (stochastic) gradient descent has an implicit bias...
research
10/17/2018

The loss surface of deep linear networks viewed through the algebraic geometry lens

By using the viewpoint of modern computational algebraic geometry, we ex...
research
05/24/2023

On progressive sharpening, flat minima and generalisation

We present a new approach to understanding the relationship between loss...
research
01/08/2022

Global Convergence Analysis of Deep Linear Networks with A One-neuron Layer

In this paper, we follow Eftekhari's work to give a non-local convergenc...
research
05/31/2022

Feature Learning in L_2-regularized DNNs: Attraction/Repulsion and Sparsity

We study the loss surface of DNNs with L_2 regularization. We show that ...

Please sign up or login with your details

Forgot password? Click here to reset