Gradient Descent Provably Optimizes Over-parameterized Neural Networks

1 Introduction

Neural networks trained by first order methods have achieved a remarkable impact on many applications, but their theoretical properties are still mysteries. One of the empirical observation is even though the optimization objective function is non-convex and non-smooth, randomly initialized first order methods like stochastic gradient descent can still find a global minimum. Surprisingly, this property is not correlated with labels. In

Zhang et al. (2016), authors replaced the true labels with randomly generated labels, but still found randomly initialized first order methods can always achieve zero training loss.

A widely believed explanation on why a neural network can fit all training labels is because the neural network is over-parameterized. For example, Wide ResNet (Zagoruyko and Komodakis, ) uses 100x parameters than the number of training data and thus there must exist one such neural network of this architecture that can fit all training data. However, the existence does not imply why the network found by a randomly initialized first order method can fit all the data. The objective function is neither smooth nor convex, which makes traditional analysis technique from convex optimization not useful in this setting. To our knowledge, only the convergence to a stationary point is known (Davis et al., 2018).

In this paper we demystify this surprising phenomenon on two-layer neural networks with rectified linear unit (ReLU) activation. Formally, we consider a neural network of the following form.

f (W, a, x) = \frac{1}{\sqrt{m}} m \sum r = 1 a_{r} σ (w_{r}^{⊤} x)

(1)

where $x \in R^{d}$ is the input, $w_{r} \in R^{d}$ is the weight vector of the first layer, $a_{r} \in R$ is the output weight and $σ (\cdot)$

is the ReLU activation function:

σ (z) = z

z \geq 0

and

σ (z) = 0

z < 0

We focus on the empirical risk minimization problem with a quadratic loss. Given a training data set ${(x_{i}, y_{i})}_{i}^{n}$ , we want to minimize

min W \in R^{d \times m} L (W) = n \sum i = 1 \frac{1}{2} {(f (W, a, x_{i}) - y_{i})}_{i}^{2} .

(2)

To do this, we fix the second layer and apply gradient descent (GD) on the first layer weights matrix

W (k + 1) = W (k) - η \frac{\partial L (W (k))}{\partial W (k)} .

(3)

where $η > 0$ is the step size. Here the gradient formula for each weight vector is ¹¹1 Note ReLU is not continuously differentiable. One can view $\frac{\partial L (W)}{\partial w_{r}}$ as a convenient notation for the right hand side of (4) and this is indeed the update rule used in practice.

\frac{\partial L (W)}{\partial w_{r}} = \frac{1}{\sqrt{m}} n \sum i = 1 (f (W, a, x_{i}) - y_{i}) a_{r} x_{i} I {w_{r}^{⊤} x_{i} \geq 0}

(4)

Though this is only a shallow fully connected neural network, the objective function is still non-smooth and non-convex due to the use of ReLU activation function. Even for this simple function, why randomly initialized first order method can achieve zero training error is not known. In fact, many previous work have tried to answer this question or similar ones. Attempts include landscape analysis (Soudry and Carmon, 2016)

, partial differential equations

(Mei et al., ), analysis of the dynamics of the algorithm (Li and Yuan, 2017), optimal transport theory (Chizat and Bach, 2018), to name a few. These results often rely strong assumptions on the labels and input distributions, or do not imply why randomly initialized first order method can achieve zero training loss. See Section 2 for detailed comparisons between our result and previous ones.

In this paper, we rigorously prove that as long as the data set is not degenerate and $m$ is large enough, with properly randomly initialized $a$ and $W (0)$ , GD achieves zero training loss at a linear convergence rate, i.e., it finds a solution $W (K)$ with $L (W (K)) \leq ϵ$ in $K = O (log (1 / ϵ))$ iterations.²²2Here we omit the polynomial dependency on $n$ and other data dependent quantities. Thus, our theoretical result not only shows the global convergence but also gives a quantitative convergence rate in terms of the desired accuracy.

Analysis Technique Overview

Our proof relies on the following insights. First we directly analyze the dynamics of each individual prediction $f (W, a, x_{i})$ for $i = 1, \dots, n$ . This is different from many previous work (Du et al., 2017b; Li and Yuan, 2017) which tried to analyze the dynamics of the parameter ( $W$ ) we are optimizing. Note because the objective function is non-smooth and non-convex, analysis of the parameter space dynamics is very difficult. In contrast, we find the dynamics of prediction space is governed by the spectral property of a Gram matrix (which can vary in each iteration, c.f. (5

)) and as long as this Gram matrix’s least eigenvalue is lower bounded, gradient descent enjoys a linear rate. Furthermore, previous work has shown in the initialization phase this Gram matrix does has lower bounded least eigenvalue as long as the data is not degenerate

(Xie et al., 2017). Thus the problem reduces to showing the Gram matrix at later iterations is close to that in the initialization phase. Our second observation is this Gram matrix is only related to the activation patterns (

I {w_{r}^{⊤} x_{i} \geq 0}

) and we can use matrix perturbation analysis to show if most of the patterns do not change, then this Gram matrix is close to its initialization. Our third observation is we find over-parameterization, random initialization and the linear convergence jointly restrict every weight vector

w_{r}

is close to its initialization. Then we can use this property to show most of the patterns do not change. Combining these insights we prove the first global quantitative convergence result of gradient descent on ReLU activated neural networks for the empirical risk minimization problem. Notably, our proof only uses linear algebra and standard probability bounds so we believe it can be easily generalized to analyze deep neural networks.

Notations

We use bold-faced letters for vectors and matrices. We Let $[n] = {1, 2, \dots, n}$ . Given a set $S$ , we use $u n i f {S}$

to denote the uniform distribution over

S

. Given an event

E

, we use

I {A}

to be the indicator on whether this event happens. We use

N (0, I)

to denote the standard Gaussian distribution. For a matrix

A

, we use

A_{i j}

to denote its

(i, j)

-th entry. We use

{∥ \cdot ∥}_{2}

to denote the Euclidean norm of a vector, and use

{∥ \cdot ∥}_{F}

to denote the Frobenius norm of a matrix. If a matrix

A

is positive semi-definite, we use

λ_{min} (A)

to denote its smallest eigenvalue. We use

⟨ \cdot, \cdot ⟩

to denote the standard Euclidean inner product between two vectors. Lastly, let

O (\cdot)

and

Ω (\cdot)

denote standard Big-O and Big-Omega notations, only hiding absolute constants.

2 Comparison with Previous Results

In this section we survey an incomplete list of previous attempts in analyzing why first order methods can find a global minimum.

Landscape Analysis

A popular way to analyze non-convex optimization problems is to identify whether the optimization landscape has some benign geometric properties. Recently, researchers found if the objective function is smooth and satisfies (1) all local minima are global and (2) for every saddle point, there exists a negative curvature, then the noise-injected (stochastic) gradient descent (Jin et al., 2017; Ge et al., 2015; Du et al., 2017a) can find a global minimum in polynomial time. This algorithmic finding encouraged researchers to study whether the deep neural networks also admit these properties.

For the objective function define in (2), some partial results were obtained. Soudry and Carmon (2016) showed if $m d \geq n$ , then at every differentiable local minimum, the training error is zero. However, since the objective is non-smooth, it is hard to show gradient descent actually convergences to a differentiable local minimum. Xie et al. (2017)

studied the same problem and related the loss to the gradient norm through the least singular value of the “extended feature matrix”

D

at the stationary points. However, they did not prove the convergence rate of gradient norm. Interestingly, our analysis relies on the Gram matrix which is actually

D D^{⊤}

Landscape analyses of ReLU activated neural networks for other settings have also been studied in many previous work (Ge et al., 2017; Safran and Shamir, 2018, 2016; Zhou and Liang, 2017; Freeman and Bruna, 2016; Hardt and Ma, 2016). These works establish favorable landscape properties such as all local minimizers are global, but do not ensure that gradient descent converges to a global minimizer of the empirical risk. For other activation functions, some previous work show the landscape does have the desired geometric properties (Du and Lee, 2018; Soltanolkotabi et al., 2018; Nguyen and Hein, 2017; Kawaguchi, 2016; Haeffele and Vidal, 2015; Andoni et al., 2014; Venturi et al., 2018). However, it is unclear how to extend their analyses in our setting.

Analysis of Algorithm Dynamics

Another way to prove convergence result is to directly analyze the dynamics of first order methods. Our paper also belongs to this category. Many previous work assumed (1) the input distribution is Gaussian and (2) the label is generated according to a planted neural network. Based on these two (unrealistic) conditions, it can be shown that randomly initialized (stochastic) gradient descent can learn a ReLU (Tian, 2017; Soltanolkotabi, 2017), a single convolutional filter (Brutzkus and Globerson, 2017)

, a convolutional neural network with one filter and one output layer

(Du et al., 2018b) and residual network with small spectral norm weight matrix (Li and Yuan, 2017).³³3Since these work assume the label is realizable, converging to global minimum is equivalent to recover the underlying model. Beyond Gaussian input distribution, Du et al. (2017b) showed for learning a convolutional filter, the Gaussian input distribution assumption can be relaxed but they still require the label is generated from an underlying true filter. Comparing with these work, our paper does not try to recover the underlying true neural network. Instead, we focus on providing theoretical justification on why randomly initialized gradient descent can achieve zero training loss, which is what we can observe and verify in practice.

The most related paper is by Li and Liang (2018) who observed that when training a two-layer full connected neural network, most of the patterns ( $I {w_{r}^{⊤} x_{i} \geq 0}$ ) do not change over iterations, which we also use to show the stability of the Gram matrix. They used this observation to obtain the convergence rate of GD on a two-layer over-parameterized neural network for the cross-entropy loss. They need number of hidden nodes $m$ scales with $p o l y (1 / ϵ)$ where $ϵ$ is the desired accuracy. Thus unless the number of hidden nodes $m \to \infty$ , they result does not imply GD can achieve zero training loss. We improve by allowing the amount of over-parameterization to be independent of the desired accuracy and show GD can achieve zero training loss. Furthermore, our proof is much simpler and more transparent so we believe it can be easily generalized to analyze other neural network architectures.

Other Analysis Approaches

Chizat and Bach (2018) used optimal transport theory to analyze continuous time gradient descent on over-parameterized models. However, their results on ReLU activated neural network is only at the formal level. Mei et al. showed the dynamics of SGD can be captured by a partial differential equation in the suitable scaling limit. They listed some specific examples on input distributions including mixture of Gaussians. However, it is still unclear whether this framework can explain why first order methods can minimize the empirical risk. Daniely (2017) built connection between neural networks with kernel methods and showed stochastic gradient descent can learn a function that is competitive with the best function in the conjugate kernel space of the network. Again this work does not imply why first order method can achieve zero training loss.

3 Continuous Time Analysis

In this section, we present our result for gradient flow, i.e., gradient descent with infinitesimal step size. In the next section, we will modify the proof and give a quantitative bound for gradient descent with positive step size. Formally, we consider the ordinary differential equation

⁴⁴4Strictly speaking, this should be differential inclusion (Davis et al., 2018) defined by :

\frac{d w_{r} (t)}{d t} = - \frac{\partial L (W (t))}{\partial w_{r} (t)}

for $r \in [m]$ . We denote $u_{i} (t) = f (W (t), a, x_{i})$ the prediction on input $x_{i}$ at time $t$ and we let $u (t) = (u_{1} (t), \dots, u_{n} (t)) \in R^{n}$ be the prediction vector at time $t$ . Our main result in this section is the following theorem.

Theorem 3.1 (Convergence Rate of Gradient Flow).

Assume for all $i \in [n]$ , ${∥ x_{i} ∥}_{2} = 1$ and $| y_{i} | \leq C$ for some constant $C$ and the matrix $H^{\infty} \in R^{n \times n}$ with $H_{i j}^{\infty} = E_{w \sim N (0, I)} [x_{i}^{⊤} x_{j} I {w^{⊤} x_{i} \geq 0, w^{⊤} x_{j} \geq 0}]$ satisfies $λ_{min} (H^{\infty}) ≜ λ_{0} > 0$ . Then if we initialize $w_{r} \sim N (0, I)$ , $a_{r} \sim u n i f [{- 1, 1}]$ for $r \in [m]$ and set the number of hidden nodes $m = Ω (\frac{n^{6}}{λ_{0}^{4}})$ , with high probability over random initialization we have

{∥ u (t) - y ∥}_{2}^{2} \leq exp (- λ_{0} t) {∥ u (0) - y ∥}_{2}^{2} .

We first discuss our assumptions. We assume ${∥ x_{i} ∥}_{2} = 1$ and $| y_{i} | \leq C$ for simplicity. We can easily modify the proof by properly scaling the initialization. The key assumption is the least eigenvalue of the matrix $H^{\infty}$ is strictly positive. Interestingly, various properties of this $H^{\infty}$ matrix has been thoroughly studied in previous work (Xie et al., 2017; Tsuchida et al., 2017). In general, unless the data is degenerate, the smallest eigenvalue of $H^{\infty}$ is strictly positive. We refer readers to Xie et al. (2017); Tsuchida et al. (2017) for a detailed characterization of this matrix.

The number of hidden nodes $m$ required is $Ω (\frac{n^{6}}{λ_{0}^{4}})$ , which depends on the number of samples $n$ and $λ_{0}$ . As will be apparent in the proof, over-parameterization, i.e., the fact $m = p o l y (n, 1 / λ_{0})$ , plays a crucial role in guaranteeing gradient descent to find the global minimum. We believe using a more refined analysis, this dependency can be further improved. Lastly, note the convergence rate is linear because ${∥ u (t) - y ∥}_{2}^{2}$ decreases to $0$ exponentially fast. The specific rate also depends on $λ_{0}$ but independent of number of hidden nodes $m$ .

3.1 Proof of Theorem 3.1

Our first step is to calculate the dynamics of each prediction.

	$\frac{d}{d t} u_{i} (t) =$	$m \sum r = 1 ⟨ \frac{\partial f (W (t), a, x_{i})}{\partial w_{r} (t)}, \frac{d w_{r} (t)}{d t} ⟩$
	$=$	$n \sum j = 1 (y_{j} - u_{j}) ⟨ m \sum r = 1 \frac{\partial f (W (t), a, x_{i})}{\partial w_{r} (t)}, \frac{\partial f (W (t), a, x_{j})}{\partial w_{r} (t)} ⟩ ≜ n \sum j = 1 (y_{j} - u_{j}) H_{i j} (t)$

where $H (t)$ is an $n \times n$ matrix with $(i, j)$ -th entry

H_{i j} (t) = \frac{1}{m} x_{i}^{⊤} x_{j} m \sum r = 1 I {x_{i}^{⊤} w_{r} (t) \geq 0, x_{j}^{⊤} w_{r} (t) \geq 0} .

(5)

With this $H$ matrix, we can write the dynamics of prediction in a compact way:

\frac{d}{d t} u (t) = H (t) (y - u (t)) .

Note $H (t)$ is a time-dependent symmetric matrix. We first analyze its property when $t = 0$ . The following lemma shows if $m$ is large then $H (0)$ has lower bounded least eigenvalue with high probability. The proof is by standard concentration bound so we defer it to appendix.

Lemma 3.1.

If $m = Ω (\frac{n^{2}}{λ_{0}^{2}} {log}^{2} (\frac{n}{δ}))$ , we have with probability at least $1 - δ$ , $λ_{min} (H (0)) \geq \frac{3}{4} λ_{0}$ .

Our second step is to show $H (t)$ is stable in terms of $W$ . Formally, the following lemma shows if $W (t)$ is close to the initialization $W (0)$ , $H (t)$ is close $H (0)$ and has a lower bounded least eigenvalue.

Lemma 3.2.

Suppose for a given time $t$ , for all $r \in [m]$ , ${∥ w_{r} (t) - w_{r} (0) ∥}_{2} \leq \frac{c λ_{0}}{n^{2}} ≜ R$ for some small positive constant $c$ . Then we have with high probability over initialization, $λ_{min} (H (t)) > \frac{λ_{0}}{2}$ .

This lemma plays a crucial in our analysis so we give the proof below.

Proof of Lemma 3.2 We define the event

A_{i r} = {\exists w : ∥ w - w_{r} (0) ∥ \leq R, I {x_{i}^{⊤} w_{r} (0) \geq 0} \neq I {x_{i}^{⊤} w \geq 0}} .

Note this event happens if and only if $∣ ∣ w_{r} (0)^{⊤} x_{0} ∣ ∣ < R$ . Recall $w_{r} (0) \sim N (0, I)$ . By anti-concentration inequality of Gaussian, we have $P (A_{i r}) = P_{z \sim N (0, 1)} (| z | < R) \leq \frac{2 R}{\sqrt{2 π}} .$ Therefore, we can bound the entry-wise deviation on $H (t)$ in expectation:

		$E [∣ ∣ H_{i j} (t) - H_{i j} (0) ∣ ∣]$
	$=$	$E [\frac{1}{m} ∣ ∣ ∣ ∣ x_{i}^{⊤} x_{j} m \sum r = 1 (I {w_{r} (0)^{⊤} x_{i} \geq 0, w_{r} (0)^{⊤} x_{j} \geq 0} - I {w_{r} (t)^{⊤} x_{i} \geq 0, w_{r} (t)^{⊤} x_{j} \geq 0}) ∣ ∣ ∣ ∣]$
	$\leq$	$\frac{1}{m} m \sum r = 1 E [I {A_{i r} \cup A_{j r}}] \leq \frac{4 R}{\sqrt{2 π}} .$

Summing over $(i, j)$ , we have $E [\sum_{(i, j) = (1, 1)}^{(n, n)} ∣ ∣ H_{i j} (t) - H_{i j} (0) ∣ ∣] \leq \frac{4 n^{2} R}{\sqrt{2 π}} .$ Thus by Markov inequality, with high probability, we have $\sum_{(i, j) = (1, 1)}^{(n, n)} ∣ ∣ H_{i j} (t) - H_{i j} (0) ∣ ∣ \leq C n^{2} R$ where $C$ is a large absolute constant. Next, we use matrix perturbation theory to bound the deviation from the initialization

{∥ H (t) - H (0) ∥}_{2} \leq {∥ H (t) - H (0) ∥}_{F} \leq (n, n) \sum (i, j) = (1, 1) ∣ ∣ H_{i j} (t) - H_{i j} (0) ∣ ∣ \leq C n^{2} R .

Lastly, we lower bound the smallest eigenvalue by plugging in $R$

λmin(H(t))≥λmin(H(0))−Cn2R≥λ02.\qed

The next lemma shows two facts if the least eigenvalue of $H$ is lower bounded. First, the loss converges to $0$ at a linear convergence rate. Second, $w_{r} (t)$ is close to the initialization for every $r \in [m]$ . This lemma clearly demonstrates the power of over-parameterization.

Lemma 3.3.

Suppose for $0 \leq s \leq t$ , $λ_{min} (H (s)) \geq \frac{λ_{0}}{2}$ . Then we have ${∥ y - u (t) ∥}_{2}^{2} \leq exp (- λ_{0} t) {∥ y - u (0) ∥}_{2}^{2}$ and for any $r \in [m]$ ,

Proof of Lemma 3.3 Recall we can write the dynamics of predictions as $\frac{d}{d t} u (t) = H (y - u (t)) .$ We can calculate the loss function dynamics

	$\frac{d}{d t} {∥ y - u (t) ∥}_{2}^{2} =$	$- 2 {(y - u (t))}^{⊤} H (t) (y - u (t))$
	$\leq$	$- λ_{0} {∥ y - u (t) ∥}_{2}^{2} .$

Thus we have $\frac{d}{d t} (exp (λ_{0} t) {∥ y - u (t) ∥}_{2}^{2}) \leq 0,$ $exp (λ_{0} t) {∥ y - u (t) ∥}_{2}^{2}$ is a decreasing function with respect to $t$ . Using this fact we can bound the loss

{∥ y - u (t) ∥}_{2}^{2} \leq exp (- λ_{0} t) {∥ y - u (0) ∥}_{2}^{2} .

Therefore, $u (t) \to y$ exponentially fast. Now we bound the gradient. Recall for $0 \leq s \leq t$ ,

	${∥ ∥ ∥ \frac{d}{d t} w_{r} (s) ∥ ∥ ∥}_{2} =$	${∥ ∥ ∥ ∥ n \sum i = 1 (y_{i} - u_{i}) \frac{1}{\sqrt{m}} a_{r} x_{i} I {w_{r} (t)^{⊤} x_{i} \geq 0} ∥ ∥ ∥ ∥}_{2}$
	$\leq$

Integrating the gradient, we can bound the distance from the initialization

∥wr(t)−wr(0)∥2≤∫t0∥∥∥ddswr(s)∥∥∥2ds≤2√n∥y−u(0)∥2√mλ0.\qed

The next lemma shows we show if $R^{'} < R$ , the conditions in Lemma 3.2 and 3.3 hold for all $t \geq 0$ . The proof is by contradiction and we defer it to appendix.

Lemma 3.4.

If $R^{'} < R$ , we have for all $t \geq 0$ , $λ_{min} (H (t)) \geq \frac{1}{2} λ_{0}$ , for all $r \in [m]$ , ${∥ w_{r} (t) - w_{r} (0) ∥}_{2} \leq R^{'}$ and ${∥ y - u (t) ∥}_{2}^{2} \leq exp (- λ_{0} t) {∥ y - u (0) ∥}_{2}^{2}$ .

Thus it is sufficient to show $R^{'} < R$ which is equivalent to $m = Ω (\frac{n^{5} {∥ y - u (0) ∥}_{2}^{2}}{λ_{0}^{4}}) .$ We bound

E [{∥ y - u (0) ∥}_{2}^{2}] = n \sum i = 1 (y_{i}^{2} + y_{i} E [f (W (0), a, x_{i})] + E [f (W (0), a, x_{i})^{2}]) = n \sum i = 1 (y_{i}^{2} + 1) = O (n) .

Thus by Markov’s inequality, we have with high probability ${∥ y - u (0) ∥}_{2}^{2} = O (n)$ . Plugging in this bound we prove the theorem. ∎

4 Discrete Time Analysis

In this section, we show randomly initialized gradient descent with a constant positive step size converges to the global minimum at a linear rate. We first present our main theorem.

Theorem 4.1 (Convergence Rate of Gradient Descent).

Assume for all $i \in [n]$ , ${∥ x_{i} ∥}_{2} = 1$ , $| y_{i} | \leq C$ for some constant $C$ and the matrix $H^{\infty} \in R^{n \times n}$ with $H_{i j}^{\infty} = E_{w \sim N (0, I)} [x_{i}^{⊤} x_{j} I {w^{⊤} x_{i} \geq 0, w^{⊤} x_{j} \geq 0}]$ satisfies $λ_{min} (H^{\infty}) ≜ λ_{0} > 0$ . If we initialize $w_{r} \sim N (0, I)$ , $a_{r} \sim u n i f [{- 1, 1}]$ for $r \in [m]$ and the number of hidden nodes $m = Ω (\frac{n^{6}}{λ_{0}^{4}})$ and we set the step size $η = O (\frac{λ_{0}}{n^{2}})$ then with high probability over the random initialization we have for $k = 0, 1, 2, \dots$

{∥ u (k) - y ∥}_{2}^{2} \leq {(1 - \frac{η λ_{0}}{2})}^{k} {∥ u (0) - y ∥}_{2}^{2} .

Theorem 4.1 shows even though the objective function is non-smooth and non-convex, gradient descent with a constant positive step size still enjoys a linear convergence rate. Our assumptions on the least eigenvalue and the number of hidden nodes are exactly the same as the theorem for gradient flow. Notably, our choice of step size is independent of number of hidden nodes $m$ in contrast to the previous work (Li and Liang, 2018).

4.1 Proof of Theorem 4.1

We prove Theorem 4.1 by induction. Our induction hypothesis is just the following convergence rate of empirical loss.

Condition 4.1.

At the $k$ -th iteration, we have ${∥ y - u (k) ∥}_{2}^{2} \leq (1 - \frac{η λ_{0}}{2})^{k} {∥ y - u (0) ∥}_{2}^{2} .$

A directly corollary of this condition is the following bound of deviation from the initialization. The proof is similar to that of Lemma 3.3 so we defer it to appendix.

Corollary 4.1.

If Condition 4.1 holds for $k^{'} = 0, \dots, k$ , then we have for every $r \in [m]$

{∥ w_{r} (k + 1) - w_{r} (0) ∥}_{2} \leq \frac{}{4 \sqrt{n} {∥ y - u (0) ∥}_{2}} \sqrt{m} λ_{0} ≜ R^{'} .

(6)

Now we show Condition 4.1 for every $k = 0, 1, \dots$ . For the base case $k = 0$ , by definition Condition 4.1 holds. Suppose for $k^{'} = 0, \dots, k$ , Condition 4.1 holds and we want to show Condition 4.1 holds for $k^{'} = k + 1$ .

Our strategy is similar to the proof of Theorem 3.1. We define the event

A_{i r} = {\exists w : ∥ w - w_{r} (0) ∥ \leq R, I {x_{i}^{⊤} w_{r} (0) \geq 0} \neq I {x_{i}^{⊤} w \geq 0}} .

where $R = \frac{c λ_{0}}{n^{2}}$ for some small positive constant $c$ . Different from gradient flow, for gradient descent we need a more refined analysis. We let $S_{i} = {r \in [m] : I {A_{i r}} = 1}$ and $S_{i}^{⊥} = [m] ∖ S_{i}$ . The following Lemma bounds the sum of sizes of $S_{i}^{⊥}$ . The proof is similar to the analysis used in Lemma 3.2. See Section A for the whole proof.

Lemma 4.1.

With high probability over initialization, we have $\sum_{i = 1}^{n} ∣ ∣ S_{i}^{⊥} ∣ ∣ \leq C m n R$ for some positive constant $C > 0$ .

Next, we calculate the difference of predictions between two consecutive iterations, analogue to $\frac{d u_{i} (t)}{d t}$ term in Section 3.

	$u_{i} (k + 1) - u (k) =$	$\frac{1}{\sqrt{m}} m \sum r = 1 a_{r} (σ (w_{r} (k + 1)^{⊤} x_{i}) - σ (w_{r} (k)^{⊤} x_{i}))$
	$=$	$\frac{1}{\sqrt{m}} m \sum r = 1 a_{r} ⎛ ⎝ σ ⎛ ⎝ {(w_{r} (k) - η \frac{\partial L (W (k))}{\partial w_{r} (k)})}_{r}^{⊤} x_{i} ⎞ ⎠ - σ (w_{r} (k)^{⊤} x_{i}) ⎞ ⎠ .$

Here we divide the right hand side into two parts. $I_{1}^{i}$ accounts for terms that the pattern does not change and $I_{2}^{i}$ accounts for terms that pattern may change.

	$I_{1}^{i} ≜$	$\frac{1}{\sqrt{m}} \sum r \in S_{i} a_{r} ⎛ ⎝ σ ⎛ ⎝ {(w_{r} (k) - η \frac{\partial L (W (k))}{\partial w_{r} (k)})}_{r}^{⊤} x_{i} ⎞ ⎠ - σ (w_{r} (k)^{⊤} x_{i}) ⎞ ⎠$
	$I_{2}^{i} ≜$

We view $I_{2}^{i}$ as a perturbation and bound its magnitude. Because ReLU is a $1$ -Lipschitz function and $| a_{r} | = 1$ , we have

To analyze $I_{1}^{i}$ , by Corollary 4.1, we know $∥ w_{r} (k) - w_{r} (0) ∥ \leq R^{'}$ and $∥ w_{r} (k) - w_{r} (0) ∥ \leq R^{'}$ for all $r \in [m]$ . Furthermore, because $R^{'} < R$ , we know $I {w_{r} (k + 1)^{⊤} x_{i} \geq 0} = I {w_{r} (k)^{⊤} x_{i} \geq 0}$ . Thus we can find a more convenient expression of $I_{1}^{i}$ for analysis

	$I_{1}^{i} =$	$- \frac{η}{m} n \sum j = 1 x_{i}^{⊤} x_{j} (u_{j} - y_{j}) \sum r \in S_{i} I {w_{r} (k)^{⊤} x_{i} \geq 0, w_{r} (k)^{⊤} x_{j} \geq 0}$
	$=$	$- η n \sum j = 1 (u_{j} - y_{j}) (H_{i j} (k) - H_{i j}^{⊥} (k))$

where $H_{i j} (k) = \frac{1}{m} \sum_{r = 1}^{m} x_{i}^{⊤} x_{j} I {w_{r} (k)^{⊤} x_{i} \geq 0, w_{r} (k)^{⊤} x_{j} \geq 0}$ is just the $(i, j)$ -th entry of a discrete version of Gram matrix defined in Section 3 and $H_{i j}^{⊥} (k) = \frac{1}{m} \sum_{r \in S_{i}^{⊥}} x_{i}^{⊤} x_{j} I {w_{r} (k)^{⊤} x_{i} \geq 0, w_{r} (k)^{⊤} x_{j} \geq 0}$ is a perturbation matrix. Let $H^{⊥} (k)$ be the $n \times n$ matrix with $(i, j)$ -th entry being $H_{i j}^{⊥} (k)$ . Using Lemma 4.1, with high probability we obtain an upper bound of the operator norm

{∥ ∥ H^{⊥} (k) ∥ ∥}_{2}^{⊥} \leq (n, n) \sum (i, j) = (1, 1) ∣ ∣ H_{i j}^{⊥} (k) ∣ ∣ \leq \frac{n \sum_{i = 1}^{n} ∣ ∣ S_{i}^{⊥} ∣ ∣}{m} \leq \frac{n^{2} m R}{m} \leq n^{2} R .

Similar to the classical analysis of gradient, we also need bound the quadratic term.

With these estimates at hand, we are ready to prove the induction hypothesis.

	${∥ y - u (k + 1) ∥}_{2}^{2} =$	${∥ y - u (k) - (u (k + 1) - u (k)) ∥}_{2}^{2}$
	$=$	${∥ y - u (k) ∥}_{2}^{2} - 2 {(y - u (k))}^{⊤} (u (k + 1) - u (k)) + {∥ u (k + 1) - u (k) ∥}_{2}^{2}$
	$=$	${∥ y - u (k) ∥}_{2}^{2} - 2 η {(y - u (k))}^{⊤} H (k) (y - u (k))$
		$+ 2 η {(y - u (k))}^{⊤} H (k)^{⊥} (y - u (k)) - 2 η {(y - u (k))}^{⊤} I_{2}$
		$+ {∥ u (k + 1) - u (k) ∥}_{2}^{2}$
	$\leq$	$(1 - η λ_{0} + 2 η n^{2} R + 2 η n^{3 / 2} R + η^{2} n^{2}) {∥ y - u (k) ∥}_{2}^{2}$
	$\leq$	$(1 - \frac{η λ_{0}}{2}) {∥ y - u (k) ∥}_{2}^{2} .$

The third equality we used the decomposition of $u (k + 1) - u (k)$ . The first inequality we used the Lemma 3.2, the bound on the step size, the bound on $I_{2}$ , the bound on ${∥ ∥ H (k)^{⊥} ∥ ∥}_{2}^{⊥}$ and the bound on ${∥ u (k + 1) - u (k) ∥}_{2}^{2}$ . The last inequality we used the bound of the step size and the bound of $R$ . Therefore Condition 4.1 holds for $k^{'} = k + 1$ . Now by induction, we prove Theorem 4.1. ∎

5 Conclusion and Discussion

In this paper we show with over-parameterization, gradient descent provable converges to the global minimum of the empirical loss at a linear convergence rate. The key proof idea is to show the over-parameterization makes Gram matrix remain positive definite for all iterations, which in turn guarantees the linear convergence. Here we list some future directions.

First, we believe the number of hidden nodes $m$ required can be reduced. For example, previous work (Soudry and Carmon, 2016) showed $m \geq \frac{n}{d}$ is enough to make all differentiable local minima global. In our setting, using advanced tools from probability and matrix perturbation theory to analyze $H (t)$ , we may able to tighten the bound.

Another direction is to prove the global convergence of gradient descent on multiple layer neural networks and convolutional neural networks. We believe our approach is still applicable because for a fixed a neural network architecture, when the number of hidden nodes is large and the initialization scheme is random Gaussian with proper scaling, the Gram matrix is also positive definite, which ensures the linear convergence of the empirical loss (at least at the begining). We believe combing our proof idea with the discovery of balancedness between layers in Du et al. (2018a) is a promising approach.

Lastly, in our paper we used the empirical loss as a potential function to measure the progress. If we use another potential function, we may able to prove the convergence rate of accelerated methods. This technique has been exploited in Wilson et al. (2016) for analyzing convex optimization. It would be interesting to bring their idea to analyze other first order methods.

Acknowledgments

We thank Wei Hu, Jason D. Lee and Ruosong Wang for useful discussions.

References

Andoni et al. (2014) Alexandr Andoni, Rina Panigrahy, Gregory Valiant, and Li Zhang. Learning polynomials with neural networks. In International Conference on Machine Learning, pages 1908–1916, 2014.
Brutzkus and Globerson (2017) Alon Brutzkus and Amir Globerson. Globally optimal gradient descent for a ConvNet with gaussian inputs. In International Conference on Machine Learning, pages 605–614, 2017.
Chizat and Bach (2018) Lenaic Chizat and Francis Bach. On the global convergence of gradient descent for over-parameterized models using optimal transport. arXiv preprint arXiv:1805.09545, 2018.
Daniely (2017) Amit Daniely. SGD learns the conjugate kernel class of the network. In Advances in Neural Information Processing Systems, pages 2422–2430, 2017.
Davis et al. (2018) Damek Davis, Dmitriy Drusvyatskiy, Sham Kakade, and Jason D Lee. Stochastic subgradient method converges on tame functions. arXiv preprint arXiv:1804.07795, 2018.
Du and Lee (2018) Simon S Du and Jason D Lee. On the power of over-parametrization in neural networks with quadratic activation. Proceedings of the 35th International Conference on Machine Learning, pages 1329–1338, 2018.
Du et al. (2017a) Simon S Du, Chi Jin, Jason D Lee, Michael I Jordan, Aarti Singh, and Barnabas Poczos. Gradient descent can take exponential time to escape saddle points. In Advances in Neural Information Processing Systems, pages 1067–1077, 2017a.
Du et al. (2017b) Simon S Du, Jason D Lee, and Yuandong Tian. When is a convolutional filter easy to learn? arXiv preprint arXiv:1709.06129, 2017b.
Du et al. (2018a) Simon S Du, Wei Hu, and Jason D Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. arXiv preprint arXiv:1806.00900, 2018a.
Du et al. (2018b) Simon S Du, Jason D Lee, Yuandong Tian, Barnabas Poczos, and Aarti Singh. Gradient descent learns one-hidden-layer cnn: Don’t be afraid of spurious local minima. Proceedings of the 35th International Conference on Machine Learning, pages 1339–1348, 2018b.
Freeman and Bruna (2016) C Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimization. arXiv preprint arXiv:1611.01540, 2016.
Ge et al. (2015) Rong Ge, Furong Huang, Chi Jin, and Yang Yuan. Escaping from saddle points $-$
online stochastic gradient for tensor decomposition.
In Proceedings of The 28th Conference on Learning Theory, pages 797–842, 2015.
Ge et al. (2017) Rong Ge, Jason D Lee, and Tengyu Ma. Learning one-hidden-layer neural networks with landscape design. arXiv preprint arXiv:1711.00501, 2017.
Haeffele and Vidal (2015) Benjamin D Haeffele and René Vidal. Global optimality in tensor factorization, deep learning, and beyond. arXiv preprint arXiv:1506.07540, 2015.
Hardt and Ma (2016) Moritz Hardt and Tengyu Ma. Identity matters in deep learning. arXiv preprint arXiv:1611.04231, 2016.
Jin et al. (2017) Chi Jin, Rong Ge, Praneeth Netrapalli, Sham M. Kakade, and Michael I. Jordan. How to escape saddle points efficiently. In Proceedings of the 34th International Conference on Machine Learning, pages 1724–1732, 2017.
Kawaguchi (2016) Kenji Kawaguchi. Deep learning without poor local minima. In Advances In Neural Information Processing Systems, pages 586–594, 2016.
Li and Liang (2018) Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data. arXiv preprint arXiv:1808.01204, 2018.
Li and Yuan (2017) Yuanzhi Li and Yang Yuan. Convergence analysis of two-layer neural networks with relu activation. In Advances in Neural Information Processing Systems, pages 597–607, 2017.
(20) Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two-layers neural networks. Proceedings of the National Academy of Sciences.
Nguyen and Hein (2017) Quynh Nguyen and Matthias Hein. The loss surface of deep and wide neural networks. In International Conference on Machine Learning, pages 2603–2612, 2017.
Safran and Shamir (2016) Itay Safran and Ohad Shamir. On the quality of the initial basin in overspecified neural networks. In International Conference on Machine Learning, pages 774–782, 2016.
Safran and Shamir (2018) Itay Safran and Ohad Shamir. Spurious local minima are common in two-layer ReLU neural networks. In International Conference on Machine Learning, pages 4433–4441, 2018.
Soltanolkotabi (2017) Mahdi Soltanolkotabi. Learning ReLus via gradient descent. In Advances in Neural Information Processing Systems, pages 2007–2017, 2017.
Soltanolkotabi et al. (2018) Mahdi Soltanolkotabi, Adel Javanmard, and Jason D Lee. Theoretical insights into the optimization landscape of over-parameterized shallow neural networks. IEEE Transactions on Information Theory, 2018.
Soudry and Carmon (2016) Daniel Soudry and Yair Carmon. No bad local minima: Data independent training error guarantees for multilayer neural networks. arXiv preprint arXiv:1605.08361, 2016.
Tian (2017) Yuandong Tian. An analytical formula of population gradient for two-layered ReLU network and its applications in convergence and critical point analysis. In International Conference on Machine Learning, pages 3404–3413, 2017.
Tsuchida et al. (2017) Russell Tsuchida, Farbod Roosta-Khorasani, and Marcus Gallagher. Invariance of weight distributions in rectified mlps. arXiv preprint arXiv:1711.09090, 2017.
Venturi et al. (2018) Luca Venturi, Afonso Bandeira, and Joan Bruna. Neural networks with finite intrinsic dimension have no spurious valleys. arXiv preprint arXiv:1802.06384, 2018.
Wilson et al. (2016) Ashia C Wilson, Benjamin Recht, and Michael I Jordan. A Lyapunov analysis of momentum methods in optimization. arXiv preprint arXiv:1611.02635, 2016.
Xie et al. (2017) Bo Xie, Yingyu Liang, and Le Song. Diverse neural network learns true target functions. In Artificial Intelligence and Statistics, pages 1216–1224, 2017.
(32) Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. NIN, 8:35–67.
Zhang et al. (2016) Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016.
Zhou and Liang (2017) Yi Zhou and Yingbin Liang. Critical points of neural networks: Analytical forms and landscape properties. arXiv preprint arXiv:1710.11205, 2017.

Appendix A Technical Proofs

Proof of Lemma 3.1.

For note every fixed $(i, j)$ pair, $H_{i j} (0)$

is an average of independent random variables. Therefore, by Hoeffding inequality, we have with probability

1 - δ^{'}

∣ ∣ H_{i j} (0) - H_{i j}^{\infty} ∣ ∣ \leq \frac{}{2 log (1 / δ^{'})} \sqrt{m} .

Setting $δ^{'} = n^{2} δ$ and applying union bound over $(i, j)$ pairs, we have for every $(i, j)$ pair with probability at least $1 - δ$

∣ ∣ H_{i j} (0) - H_{i j}^{\infty} ∣ ∣ \leq \frac{}{4 log (n / δ)} \sqrt{m} .

Thus we have

Thus if $m = Ω (\frac{n^{2} {log}^{2} (n / δ)}{λ_{0}^{2}})$ we have the desired result. ∎

Proof of Lemma 3.4.

Suppose the conclusion does not hold at time $t$ . If there exists $r \in [m]$ , $∥ w_{r} (t) - w_{r} (0) ∥ \geq R^{'}$ or ${∥ y - u (t) ∥}_{2}^{2} > exp (- λ_{0} t) {∥ y - u (0) ∥}_{2}^{2}$ , then by Lemma 3.3 we know there exists $s \leq t$ such that $λ_{min} (H (s)) < \frac{1}{2} λ_{0}$ . By Lemma 3.2 we know there exists

t_{0} = inf {t > 0 : max r \in [m] {∥ w_{r} (t) - w_{r} (0) ∥}_{r}^{2} \geq R} .

Thus at $t_{0}$ , there exists $r \in [m]$ , ${∥ w_{r} (t_{0}) - w_{r} (0) ∥}_{r}^{2} = R$ . Now by Lemma 3.2, we know $H (t_{0}) \geq \frac{1}{2} λ_{0}$ for $t^{'} \leq t_{0}$ . However, by Lemma 3.3, we know ${∥ w_{r} (t_{0}) - w_{r} (0) ∥}_{2} < R^{'} < R$ . Contradiction.

For the other case, at time $t$ , $λ_{min} (H (t)) < \frac{1}{2} λ_{0}$ we know we know there exists

t_{0} = inf {t \geq 0 : max r \in [m] {∥ w_{r} (t) - w_{r} (0) ∥}_{r}^{2} \geq R} .

The rest of the proof is the same as the previous case. ∎

Proof of Corollary 4.1.

We use the norm of gradient to bound this distance.

	${∥ w_{r} (k + 1) - w_{r} (0) ∥}_{2} \leq$	$η k \sum k^{'} = 0 {∥ ∥ ∥ \frac{\partial L (W (k^{'}))}{\partial w_{r} (k^{'})} ∥ ∥ ∥}_{2}$
	$\leq$	$η k \sum k^{'} = 0 \frac{\sqrt{n} {∥ y - u (k^{'}) ∥}_{2}^{'}}{\sqrt{m}}$
	$\leq$	$η k \sum k^{'} = 0 \frac{\sqrt{n} (1 - \frac{η λ}{2})^{k^{'} / 2}}{\sqrt{m}} {∥ ∥ y - u (k^{'}) ∥ ∥}_{2}^{'}$
	$\leq$	$η \infty \sum k^{'} = 0 \frac{\sqrt{n} (1 - \frac{η λ_{0}}{2})^{k^{'} / 2}}{\sqrt{m}} {∥ ∥ y - u (k^{'}) ∥ ∥}_{2}^{'}$
	$=$	$\frac{4 \sqrt{n} {∥ y - u (0) ∥}_{2}}{\sqrt{} m λ_{0}} .$

∎

Proof of Lemma 4.1.

For a fixed $i \in [n]$ and $r \in [m]$ , by anti-concentration inequality, we know $P (A_{i r}) \leq \frac{2 R}{\sqrt{2 π}}$ . Thus we can bound the size of $S_{i}^{⊥}$ in expectation.

E [∣ ∣ S_{i}^{⊥} ∣ ∣] = m \sum r = 1 P (A_{i r}) \leq \frac{2 m R}{\sqrt{2 π}} .

(7)

Summing over $i = 1, \dots, n$ , we have

E [n \sum i = 1 ∣ ∣ S_{i}^{⊥} ∣ ∣] \leq \frac{2 m n R}{\sqrt{2 π}} .

Thus by Markov’s inequality, we have with high probability

n \sum i = 1 ∣ ∣ S_{i}^{⊥} ∣ ∣ \leq C m n R .

(8)

for some large positive constant $C > 0$ . ∎

Gradient Descent Provably Optimizes Over-parameterized Neural Networks

Authors

On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widths

Learning One-hidden-layer neural networks via Provable Gradient Descent with Random Initialization

A Dynamical View on Optimization Algorithms of Overparameterized Neural Networks

Implicit Regularization of Normalization Methods

A Convergence Theory for Deep Learning via Over-Parameterization

Globally Optimal Training of Generalized Polynomial Neural Networks with Nonlinear Spectral Methods

Gradient Layer: Enhancing the Convergence of Adversarial Training for Generative Models

1 Introduction

Analysis Technique Overview

Notations

2 Comparison with Previous Results

Landscape Analysis

Analysis of Algorithm Dynamics

Other Analysis Approaches

3 Continuous Time Analysis

Theorem 3.1 (Convergence Rate of Gradient Flow).

3.1 Proof of Theorem 3.1

Lemma 3.1.

Lemma 3.2.

Lemma 3.3.

Lemma 3.4.

4 Discrete Time Analysis

Theorem 4.1 (Convergence Rate of Gradient Descent).

4.1 Proof of Theorem 4.1

Condition 4.1.

Corollary 4.1.

Lemma 4.1.

5 Conclusion and Discussion

Acknowledgments

References

Appendix A Technical Proofs

Proof of Lemma 3.1.

Proof of Lemma 3.4.

Proof of Corollary 4.1.

Proof of Lemma 4.1.

Gradient Descent Provably Optimizes Over-parameterized Neural Networks

Authors

On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widths

Learning One-hidden-layer neural networks via Provable Gradient Descent with Random Initialization

A Dynamical View on Optimization Algorithms of Overparameterized Neural Networks

Implicit Regularization of Normalization Methods

A Convergence Theory for Deep Learning via Over-Parameterization

Globally Optimal Training of Generalized Polynomial Neural Networks with Nonlinear Spectral Methods

Gradient Layer: Enhancing the Convergence of Adversarial Training for Generative Models

This week in AI

1 Introduction

Analysis Technique Overview

Notations

2 Comparison with Previous Results

Landscape Analysis

Analysis of Algorithm Dynamics

Other Analysis Approaches

3 Continuous Time Analysis

Theorem 3.1 (Convergence Rate of Gradient Flow).

3.1 Proof of Theorem 3.1

Lemma 3.1.

Lemma 3.2.

Lemma 3.3.

Lemma 3.4.

4 Discrete Time Analysis

Theorem 4.1 (Convergence Rate of Gradient Descent).

4.1 Proof of Theorem 4.1

Condition 4.1.

Corollary 4.1.

Lemma 4.1.

5 Conclusion and Discussion

Acknowledgments

References

Appendix A Technical Proofs

Proof of Lemma 3.1.

Proof of Lemma 3.4.

Proof of Corollary 4.1.

Proof of Lemma 4.1.