Explorations on high dimensional landscapes

12/20/2014
by   Levent Sagun, et al.
0

Finding minima of a real valued non-convex function over a high dimensional space is a major challenge in science. We provide evidence that some such functions that are defined on high dimensional domains have a narrow band of values whose pre-image contains the bulk of its critical points. This is in contrast with the low dimensional picture in which this band is wide. Our simulations agree with the previous theoretical work on spin glasses that proves the existence of such a band when the dimension of the domain tends to infinity. Furthermore our experiments on teacher-student networks with the MNIST dataset establish a similar phenomenon in deep networks. We finally observe that both the gradient descent and the stochastic gradient descent methods can reach this level within the same number of steps.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/01/2022

Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks

Despite the non-convex optimization landscape, over-parametrized shallow...
research
05/19/2014

On the saddle point problem for non-convex optimization

A central challenge to many fields of science and engineering involves m...
research
03/08/2021

Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem

In this paper we investigate how gradient-based algorithms such as gradi...
research
04/04/2022

Deep learning, stochastic gradient descent and diffusion maps

Stochastic gradient descent (SGD) is widely used in deep learning due to...
research
08/13/2022

Differentiable Inductive Logic Programming in High-Dimensional Space

Synthesizing large logic programs through Inductive Logic Programming (I...
research
07/28/2023

On Single Index Models beyond Gaussian Data

Sparse high-dimensional functions have arisen as a rich framework to stu...
research
08/05/2023

K-band: Self-supervised MRI Reconstruction via Stochastic Gradient Descent over K-space Subsets

Although deep learning (DL) methods are powerful for solving inverse pro...

Please sign up or login with your details

Forgot password? Click here to reset