An Essay on Optimization Mystery of Deep Learning

by   Eugene Golikov, et al.

Despite the huge empirical success of deep learning, theoretical understanding of neural networks learning process is still lacking. This is the reason, why some of its features seem "mysterious". We emphasize two mysteries of deep learning: generalization mystery, and optimization mystery. In this essay we review and draw connections between several selected works concerning the latter.




This emphasizes two mysteries of deep learning: generalization mystery, and optimization mystery. In this essay, get more info and draw connections between several selected works concerning the latter. I got it was stunning and had full things.


This emphasizes two mysteries of deep learning: generalization mystery, and optimization mystery. In this essay, get more info and draw connections between several selected works concerning the latter. I got it was stunning and had full things.


page 1

page 2

page 3

page 4


How deep learning works --The geometry of deep learning

Why and how that deep learning works well on different tasks remains a m...

The Modern Mathematics of Deep Learning

We describe the new field of mathematical analysis of deep learning. Thi...

Comparative analysis of SVIT and Skype features in e-Learning process

The article discusses capabilities of the SVIT and Skype programs in ter...

Neurashed: A Phenomenological Model for Imitating Deep Learning Training

To advance deep learning methodologies in the next decade, a theoretical...

Regression with Deep Learning for Sensor Performance Optimization

Neural networks with at least two hidden layers are called deep networks...

Winning with Simple Learning Models: Detecting Earthquakes in Groningen, the Netherlands

Deep learning is fast emerging as a potential disruptive tool to tackle ...

Model Complexity of Deep Learning: A Survey

Model complexity is a fundamental problem in deep learning. In this pape...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Despite its huge empirical success, deep learning still preserves many features of alchemy [Rahimi, 2017]: progress in this field is obtained mainly by trial and error, and our intuition about how do neural networks actually work often misleads us.

Alchemy, in order to become usual chemistry, needs a theoretical ground. For now, a solid theoretical ground for deep learning is lacking, however, fortunately, many pieces of theory appeared from different directions during several past years. The purpose of this essay is not to provide a comprehensive review, but to draw connections between some works on this topic. The list of works mentioned here is by no means representative, or, all the more so, complete.

Since the theory of deep learning is lacking, some features of neural networks learning seem ”mysterious”. We emphasize two mysteries of deep learning:

  1. Generalization mystery. It is very common for contemporary neural networks to have many more parameters than the number of training examples at hand. Having huge abundance of parameters results in existence of ”bad” local minima in terms of test error [Zhang et al., 2016]

    . Classic generalization bounds based on VC-dimension are pessimistic, and consequently vacuous in this case. However, stochastic gradient descent seems to avoid these bad local minima. Hence new, neural-net-specific, bounds have to be developed.

  2. Optimization mystery. It is indeed surprising, why (stochastic) gradient descent does not get trapped into ”bad” local minima in terms of test error. However, it is also quite surprising, why gradient descent (a local optimization method!) does not get trapped into bad local minima in terms of train error. It seems true that, since the number of parameters in modern neural networks is usually much greater than the number of training samples, there exists such parameter configuration for which our neural net perfectly fits the data, however, it is far from obvious why the gradient descent finds this configuration.

In the current essay we will talk about the optimization mystery.

Figure 1: Results of training Conv-small of [Miyato et al., 2017] on subset of 4000 samples of CIFAR10. Initial numbers of filters in convolutional layers were multiplied by width factor.

In order to illustrate the above-mentioned phenomena, we trained a modern convolutional architecture, Conv-small from [Miyato et al., 2017], on a subset of 4000 examples of CIFAR-10 dataset. We also multiplied the number of filters in each convolutional layer by some width factor, in order to illustrate how the number of parameters affects optimization and generalization.

The results are shown in Figure 1. One can see that the network indeed generalizes well, despite of severe overparameterization. Moreover, the test accuracy does not decrease as network grows larger. From the optimization side, we see that SGD indeed finds a global minimum in terms of train error (note that there are no global minima in terms of cross-entropy in any finite ball in weight-space). Moreover, as network grows larger, optimization becomes faster (in terms of number of iterations).

2 Optimization mystery

2.1 Loss landscape

Let us first deal with one of these observations: gradient descent finds a global minimum for random initialization. It is known for randomly initialized gradient descent that it is able to find a local minimum almost surely (Theorem 4.8 of [Lee et al., 2016]). Hence it is tempting to propose the following hypothesis:

All local minima of loss landscape of neural nets are global. (1)

We begin with the simplest optimization problem — -regression:


One can easily show that all local minima of this problem are indeed global, and in the case of non-degenerate data covariance matrix , this minimum is unique.

There are two possible ways to depart from this trivial setting: acquiring depth, and adding a non-linearity.

2.1.1 Deep linear nets

Let us consider a deep linear net first. The corresponding -regression problem is given as follows:


This setting looks odd, since in terms of a class of realizable functions, a deep linear net is equivalent to a shallow one with rank restriction. Indeed, let

, and consider the following shallow problem:


where — index of the ”bottleneck” layer. Then, we have a many-to-one correspondence:

and all minima of a deep problem are of the same value as ones of a shallow problem:

Note, however, that both problems are (generally) non-convex: shallow problem is non-convex, because of non-convexity of its optimization domain (for non-trivial rank-constraint), and deep problem is non-convex due to weight-space symmetries. Indeed, if we multiply one of the weight matrices by a non-zero factor, and divide another weight matrix by the same factor simultaneously, the corresponding function will not change.

The above-proposed hypothesis 1 is false for a general non-convex problem, however, it was proven in  [Lu and Kawaguchi, 2017] (see Theorem 2.2 there) that it is true for a rank-constrained shallow problem111More precisely, Theorem 2.2 of [Lu and Kawaguchi, 2017] assumes a finite dataset. It is not obvious how to generalize it to a general data distribution.. Moreover, [Lu and Kawaguchi, 2017] prove that the same is true for a deep problem too.

Consider a finite dataset of size . Assume to be of full rank (this corresponds to non-degeneracy of the corresponding covariance matrix). At the high level, the proof is based on three theorems:

  1. Theorem 2.1: If is a local minimum of , then is a local minimum of .

  2. Theorem 2.2: Every local minimum of is global.

  3. Theorem 2.3: Every local minimum of is global.

Note that Theorem 2.3 is a simple corollary of two previous theorems. Indeed, let be a local minimum of , then due to Theorem 2.1, is a local minimum of . Since all local minima of are of the same value (Theorem 2.2), all local minima of are of the same value too. Hence they are all global.

Proof of Theorem 2.1 starts with the following observation. Let be a local minimum of . Let be a small perturbation of . In order to prove that is a local minimum of , we have to show that for any perturbation of holds . Our aim then is to prove that for any perturbation there exists a perturbation such that . Once this is proven, we can conclude that:

Perturbation that corresponds to perturbation is explicitly constructed for the case , and . Generalization to is given by induction (Theorem 3.1). If , it is proven in Theorem 3.2 and Theorem 3.3 that one can perturb in such a way that perturbation would have , will be a local minimum of , and loss will not change. In other words, we can always perturb a local minimum to make it full rank and retain local minimality.

2.1.2 Non-linear nets

So, our hypothesis is true for linear nets. However, a linear net is not a practical case, and we have to move to non-linear nets. A general loss of the simplest non-linear net is given as follows:



is an element-wise non-linearity. Unfortunately, it happens that even a simple one-layer non-linear regression has a quite hard-to-analyze loss landscape, even for gaussian data distribution and labels generated from a teacher net of the same architecture (see, e.g. 

[Tian, 2017]). Hence one can hardly hope to obtain a simple, analyzable loss landscape for a non-linear net with one hidden layer:

Wide shallow sigmoid nets.

Surprisingly, our hypothesis becomes true for a non-linear net as its hidden layer becomes sufficiently wide. More precisely, consider the following variant of the previous problem:


where index denotes Frobenius norm, is a finite dataset of size , . Note that here denotes the width of a hidden layer. [Yu and Chen, 1995] proved that as long as and columns of (data points) are distinct, for all local minima are global222Actually, [Yu and Chen, 1995] proved their theorem for the case , but it is not hard to generalize it to ..

The proof is based on the following observation. Let be a local minimum of 7. Let us fix , and consider the following surrogate loss:

Note that corresponds to the loss of a linear regression 2 with modified dataset . Since the problem is convex, all of its critical points are minima of the same value. Obviously, if has rank (this necessitates the width of a hidden layer to be greater than the number of datapoints ), we can perfectly fit the modified dataset, and all local minima of have zero value. In this case, , and is indeed a global minimum of .

Unfortunately, the condition is not sufficient for to always be of rank . However, it is proven in the paper that as long as all columns of are distinct, the set of such for which , has Lebesgue measure of zero333[Yu and Chen, 1995] gave an incorrect proof of this statement. One can find a correct proof of a more general statement in Lemma 4.4 of [Nguyen and Hein, 2017]

. The proof sufficiently utilizes the fact that sigmoid is an analytic function; the proven statement is false for ReLU.

The proof of the main theorem proceeds as follows. We have already proven the theorem for . However, if , since the value of the measure for the corresponding is zero, we can find a small enough perturbation , so that . Suppose . Then, for a small enough perturbation , . Launch a gradient flow on . It will find a point with zero loss, since . Hence a gradient flow leaves some fixed vicinity of (vicinity of positive loss), no matter how small the perturbation is. This means that the point is unstable in Lyapunov sense, hence it cannot be a local minimum. Hence all local minima of 7 have value zero, regardless of the rank of ; hence all of them are global.

Deep and wide sigmoid nets.

We see that our hypothesis 1 is true for wide-enough shallow nets. What about deep nets? Consider a problem, similar to 7:


where as before, denotes a finite dataset of size , , and is an element-wise sigmoid non-linearity. In the previous theorem the main assumption was that the hidden layer was wide enough. Here we assume a similar restriction:

These two assumptions are enough to prove that the set of , such that has rank , has Lebesgue measure zero (Lemma 4.4 of [Nguyen and Hein, 2017]).

Let us try directly applying the logic of the previous theorem. For a shallow net implied at a local minimum. This was due to the fact that the problem was convex (since it was a linear regression). Unfortunately, given , the problem is not generally convex (since for it is not a linear regression any more). Hence it is not true that all local minima have zero loss, even given .

However, if we additionally assume that our local minimum is non-degenerate in the following sense:

then at a local minimum will imply that the loss in this local minimum is zero (Lemma 3.5 of [Nguyen and Hein, 2017]). Note that for a shallow net , , and this assumption is trivially true. Note also that this assumption implies . In other words, the width of a net should not increase after the -th layer.

The condition does not generally hold even given . Similar to the theorem concerning shallow nets, we have to deal with the case when at our local minimum. Previously, we have used the convexity of the problem , which implied that continuous-time gradient descent finds a global minimum of this problem. Now we have the problem , which is not generally convex, and we do not have a guarantee to find a global minimum of it with continuous-time gradient descent any more.

In their Theorem 4.6, [Nguyen and Hein, 2017] use the following technique. They assume the Hessian to be non-degenerate at a local minimum we consider. This allows them to use the implicit function theorem to deduce that if is a critical point, then for any perturbation one can find a perturbation , such that is also a critical point. Take a perturbation such that has rank (such perturbation exists due to Lemma 4.4, mentioned previously). The corresponding perturbed matrices for will have ranks as long as perturbations are small enough. Hence at the perturbed point we find ourselves at a simple, previously-discussed case, and conclude that

. However, since the loss function is continuous wrt weights, we further conclude that

, and the considered minimum is global.

To sum up, the main theorem (Corollary 3.9 of [Nguyen and Hein, 2017]) states the following. Let be a local minimum of a problem 8. Assume the following conditions hold:

  1. Columns of (data points) are distinct;

  2. One of the layers is wide enough: ;

  3. Our minimum is non-degenerate: ;

  4. Loss function has non-degenerate Hessian at our minimum: .

Then, , and is a global minimum of 8.

Unfortunately, as we see, this theorem does not state that all local minima of 8 are global. It states only that some minima, which are non-degenerate in some sense, are global. We hypothesize that the third condition could be relaxed to simply , but we also suspect that degenerate minima in terms of the Hessian can exist, and our initial hypothesis 1 is generally false.

Remark on non-smooth non-linearities.

The above theorem could be generalized to all commonly-used smooth non-linearities, e.g. tanh, softplus, however, it is unlikely for it to be generalized to non-smooth ones: ReLU, or LeakyReLU. Indeed, when some units of a ReLU network are deactivated, we effectively obtain a smaller network. We do not have guarantees for not-wide-enough nets. Hence, in the worst case, a smaller network could have local minima. These local minima could become (degenerate) saddles in the initial network, or again (degenerate) local minima. It is not obvious, whether the second alternative is possible or not. The same reasoning holds for the theorem of [Yu and Chen, 1995].

Remark on criticality with respect to .

One can notice that in the proof of the theorem of [Nguyen and Hein, 2017] we have never used the fact that the local minimum is a critical point of loss function wrt the first weight matrices, i.e. that . It means that given the four above-stated conditions, we only need to assume to be a local minimum of . Hence, according to the theorem, becomes a minimum of automatically. It is worth thinking of how the condition could be used to relax some of the four above-stated assumptions. The same reasoning holds for theorem of [Yu and Chen, 1995].

2.2 Gradient descent dynamics

As we have seen in the previous section, despite the fact, that gradient descent is guaranteed to converge to a local minimum, it is not obvious, whether all local minima are global or not, especially for ReLU networks, where theorem of [Nguyen and Hein, 2017] does not hold. However, as we can see in Figure 1, (stochastic) gradient descent indeed converges to a global minimum on the train set.

A prominent result in this direction was obtained by [Du et al., 2019]. Consider a ReLU net with one hidden layer and a single output:


where is an input, are weights of the hidden layer, are weights of the output layer, and denotes an element-wise ReLU. Note that the width of the hidden layer is denoted by here. We consider regression on a finite dataset of size . This leads to the following loss function:


Note that the corresponding optimization problem is very similar to 7. Our new problem differs significantly from 7 in the following aspects:

  1. ReLU instead of sigmoid,

  2. optimization is performed wrt to only.

Note also the change of notation (in order to be compatible with the paper).

The result of [Du et al., 2019] easily transfers to sigmoid, but it is unlikely that the result of [Yu and Chen, 1995] could be generalized to ReLU. Hence the first point is by no means a drawback, it is an advantage.

The last point is even more crucial. The proof of theorem of [Yu and Chen, 1995] critically relied on optimization wrt (which is denoted as here). The proof of [Du et al., 2019] does not rely on it. The main result (which we have not stated yet), could be obtained with or without optimization wrt .

Before stating the main result of [Du et al., 2019], we need several assumptions and definitions. Assume that the weights are initialized as follows:

Assume also that datapoints lie on a sphere and y’s are bounded:

We consider dynamics of a continuous-time gradient descent:

In the proof we are going to reason about the dynamics of network individual predictions:

We now proceed to the main result. Theorem 3.2 of [Du et al., 2019] states the following. Let and ; then w.p. over initialization we have:


Here is a data-dependent constant, which we will define soon. Theorem 3.1 of [Du et al., 2019] states that as long as there are no data points which are parallel to each other, this constant is strictly positive.

Let us take a closer look at this result. It states basically that as long as the hidden layer is wide enough, neural network predictions on data-points converge to ground-truth answers exponentially fast with high probability. A trivial consequence is that continuous-time gradient descent indeed converges to global minimum with high probability.

Note that the continuous-time gradient descent is assumed only for convenience. A similar result (with proper learning rate) could be obtained for discrete-time gradient descent as well (Theorem 4.1 of [Du et al., 2019]).

Proof sketch.

The proof starts with the following reasoning. We want to know how network predictions change with time:

where square brackets denote indicators. Note that at initialization, and does not change during training process. Hence we shall omit it from now on. If we define


then we can rewrite the previous equation as:

This differential equation governs the evolution of network predictions on train set. One can easily show that

is a Gram matrix, hence it is positive semi-definite. Our main goal is to show that its minimal eigenvalue remains bounded below with

throughout the whole process of optimization:

Our goal is to prove . (13)

If this holds, we easily obtain the desired result:

After solving the differential inequality we get:

which is 11. Before giving the sketch of the proof, we have to define . Let be the expectation of Gram matrix at initialization :

We define as its minimal eigenvalue, which is non-negative, since is a Gram matrix. Theorem 3.1 of [Du et al., 2019] states that it is positive as long as data-points are not parallel to each other.

The proof of 13 is divided into two parts. In the first part we prove that as long as is large enough, is close to its expectation, , in terms of norm. As a consequence, the spectrum of does not differ much from the spectrum of , and . In the second part we prove that as long as is large enough, does not change much in terms of norm throughout the optimization process. As a consequence, its spectrum also does not change much, and .

The first part is covered by Lemma 3.1 of [Du et al., 2019]. Essentially, it uses a concentration bound (Hoeffding inequality) to bound , and then to bound .

The second part is a bit more involved. Lemma 3.2 of [Du et al., 2019] states that as long as weights are close enough to initialization, Gram matrix does not change much. This is due to the fact that if we perturb the initial weights, few of the indicators in the definition 12 change. Lemma 3.3 states that as long as on , is sufficiently close to . Lemmas 3.2 and 3.3 together imply that on is equivalent to the fact that is close to on . Since for both statements should necessarily be true, Lemma 3.4 states that they are true for all using continuity argument.

Remark on convergence rate.

The theorem 11 states that neural network predictions on train dataset converge to train labels exponentially fast, which is a very strong result. However, in optimization theory convergence rate is usually measured in terms of how fast the algorithm converges to a stationary point. A (first-order) stationary point is a critical point of loss function:

A second-order stationary point is a stationary point with positive semi-definite hessian of loss function:

It is well known in optimization theory that given a general gradient-Lipschitz loss function gradient descent converges to a stationary point in time independent from the number of parameters . Let us look at how fast the gradient of loss function decays to zero in our case:

We see that as increases (which is related to the number of parameters), gradient norm decreases, hence the time to reduce the gradient norm to a value below some epsilon also decreases, which was not the case in general setting.

What about second-order stationary points? Theorem 4.8 of [Lee et al., 2016] states that randomly initialized gradient descent converges to a second-order stationary point almost surely. However, this theorem tells nothing about the convergence rate. [Du et al., 2017] constructs an example of loss function for which randomly initialized gradient descent requires time exponential in in order to converge to a second-order stationary point. It is not hard to check that in our case the Hessian of the loss function is positive semi-definite almost everywhere (except the points where ReLU is not differentiable). Indeed, let us take a look at our Hessian:

We see that the Hessian is a sum of Gram matrices, which are positive semi-definite, hence the whole Hessian is positive semi-definite too. Given that, our method converges to a second-order stationary point in time that decreases with the number of parameters.

3 Conclusion

As we have seen, the optimization mystery could be revealed, but either for linear networks (theorem of [Lu and Kawaguchi, 2017]), which are not commonly used, or for unrealistically wide non-linear nets (theorems of [Yu and Chen, 1995], [Nguyen and Hein, 2017], and [Du et al., 2019]). We suspect that assuming a cluster structure of the dataset (i.e. classes of MNIST are separated from each other in pixel space) will allow to reduce the required amount of overparameterization sufficiently (this idea was actually elaborated in [Nguyen and Hein, 2017]). However, it seems that even for unstructured data, the amount of overparameterization required in theorem of  [Du et al., 2019] is still too large [Zhang et al., 2016]. We suspect that these bounds could be reduced even in general case using more involved techniques.


  • [Du et al., 2017] Du, S. S., Jin, C., Lee, J. D., Jordan, M. I., Poczos, B., and Singh, A. (2017). Gradient Descent Can Take Exponential Time to Escape Saddle Points. arXiv e-prints.
  • [Du et al., 2019] Du, S. S., Zhai, X., Poczos, B., and Singh, A. (2019). Gradient descent provably optimizes over-parameterized neural networks. In International Conference on Learning Representations.
  • [Lee et al., 2016] Lee, J. D., Simchowitz, M., Jordan, M. I., and Recht, B. (2016). Gradient Descent Converges to Minimizers. arXiv e-prints.
  • [Lu and Kawaguchi, 2017] Lu, H. and Kawaguchi, K. (2017). Depth Creates No Bad Local Minima. arXiv e-prints.
  • [Miyato et al., 2017] Miyato, T., Maeda, S.-i., Koyama, M., and Ishii, S. (2017).

    Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning.

    arXiv e-prints.
  • [Nguyen and Hein, 2017] Nguyen, Q. and Hein, M. (2017). The loss surface of deep and wide neural networks. arXiv e-prints.
  • [Rahimi, 2017] Rahimi, A. (2017). Machine Learning has become Alchemy. Presented at NIPS2017, Test of Time Award.
  • [Tian, 2017] Tian, Y. (2017). An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysis. arXiv e-prints.
  • [Yu and Chen, 1995] Yu, X.-H. and Chen, G.-A. (1995).

    On the Local Minima Free Condition of Backpropagation Learning.

    IEEE Transactions on Neural Networks, pages 1300 – 1303.
  • [Zhang et al., 2016] Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. (2016). Understanding deep learning requires rethinking generalization. arXiv e-prints.