Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks

1 Introduction

Despite the extensive empirical success of deep networks, their optimization and generalization properties are still not well understood. Recently, the neural tangent kernel (NTK) has provided the following insight into the problem. In the infinite-width limit, the NTK converges to a limiting kernel which stays constant during training; on the other hand, when the width is large enough, the function learned by gradient descent follows the NTK (Jacot et al., 2018)

. This motivates the study of overparameterized networks trained by gradient descent, using properties of the NTK. In fact, parameters related to NTK, such as the minimum eigenvalue of the limiting kernel, appear to affect optimization and generalization

(Arora et al., 2019).

However, in addition to such NTK-dependent parameters, prior work also requires the width to depend polynomially on $n$ , $1 / δ$ or $1 / ϵ$ , where $n$ denotes the size of the training set, $δ$ denotes the failure probability, and $ϵ$ denotes the target error. These large widths far exceed what is used empirically, constituting a significant gap between theory and practice.

Our contributions.

In this paper, we narrow this gap by showing that a two-layer ReLU network with $Ω (ln (n / δ) + ln (1 / ϵ)^{2})$ hidden units trained by gradient descent achieves classification error $ϵ$ on test data, meaning both optimization and generalization occur. Unlike prior work, the width is fully polylogarithmic in $n$ , $1 / δ$ , and $1 / ϵ$ ; the width will additionally depend on the separation margin of the limiting kernel, a quantity which is guaranteed positive (assuming no inputs are duplicated with noisy labels), and can distinguish between true labels and random labels. The paper organization together with some details are described below.

Section 2: studies gradient descent on the training set. Using the $ℓ_{1}$ geometry inherent in classification tasks, we prove that with any width at least polylogarithmic and any constant step size no larger than $1$ , gradient descent achieves training error $\eps$ in $˜ O (1 / ϵ)$ iterations (cf. Section 2). As is common in the NTK literature (Chizat and Bach, 2019), we also show the parameters hardly change, which will be essential to our generalization analysis.
Section 3: gives a test error bound. Concretely, using the preceding gradient descent analysis, and standard Rademacher tools but exploiting how little the weights moved, we show that with $˜ Ω (1 / ϵ^{2})$ samples and $˜ O (1 / ϵ)$ iterations, gradient descent finds a solution with $ϵ$ test error (cf. Section 3 and Section 3). (As discussed in Section 3, $˜ Ω (1 / ϵ)$ samples also suffice via a smoothness-based generalization bound, at the expense of large constant factors.)
Section 4: considers stochastic gradient descent (SGD) with access to a standard stochastic online oracle. We prove that with width at least polylogarithmic and sample complexity
$˜ O (1 / ϵ)$ , SGD achieves an arbitrarily small test error (cf. Section 4).
Section 5: discusses the separability condition, which is in general a positive number, but reflects the difficulty of the classification problem. Regarding random labels, we show that starting from a distribution with a good positive margin, but replacing the labels with random noise, the margin can degrade all the way down to $O (1 / \sqrt{n})$ , which (correctly) removes the possibility of generalization. In this way, our analysis can distinguish between true labels and random labels.
Section 6: concludes with some open problems.

1.1 Related work

There has been a large literature studying gradient descent on overparameterized networks via the NTK. The most closely related work is (Nitanda and Suzuki, 2019), which shows that a two-layer network trained by gradient descent with the logistic loss can achieve a small test error, under the same assumption that the neural tangent model with respect to the first layer can separate the data distribution. However, they analyze smooth activations, while we handle the ReLU. They require $Ω (1 / ϵ^{2})$ hidden units, $˜ Ω (1 / ϵ^{4})$ data samples, and $O (1 / ϵ^{2})$ steps, while our result only needs polylogarithmic hidden units, $˜ Ω (1 / ϵ^{2})$ data samples, and $˜ O (1 / ϵ)$ steps.

Additionally on shallow networks, Du et al. (2018b) prove that on an overparameterized two-layer network, gradient descent can globally minimize the empirical risk with the squared loss. Their result requires $Ω (n^{6} / δ^{3})$ hidden units. Oymak and Soltanolkotabi (2019) further reduces the required overparameterization, but it still has a $\poly(n)$ dependency. Using the same amount of overparameterization as (Du et al., 2018b), Arora et al. (2019)

further show that the two-layer network learned by gradient descent can achieve a small test error, assuming that on the data distribution the smallest eigenvalue of the limiting kernel is at least some positive constant. They also give a fine-grained characterization of the predictions made by gradient descent iterates; such a characterization makes use of a special property of the squared loss and cannot be applied to the logistic regression setting.

Li and Liang (2018) show that stochastic gradient descent (SGD) with the cross entropy loss can learn a two-layer network with small test error, using

\poly(ℓ,1/ϵ)

hidden units, where

ℓ

is at least the covering number of the unit sphere using balls whose radii are no larger than the smallest distance between two data points with different labels. In a high-dimensional space,

ℓ

could be very large. Allen-Zhu et al. (2018a) consider SGD on a two-layer network, and a variant of SGD on a three-layer network. The three-layer analysis further exhibits some properties not captured by the NTK. They assume a ground truth network with infinite-order smooth activations, and they require the width to depend polynomially on

1 / ϵ

and some constants related to the smoothness of the activations of the ground truth network.

On deep networks, a variety of works have established low training error (Allen-Zhu et al., 2018b; Du et al., 2018a; Zou et al., 2018; Zou and Gu, 2019). Cao and Gu (2019a) assume that the neural tangent model with respect to the second layer of a two-layer network can separate the data distribution, and prove that gradient descent on a deep network can achieve $ϵ$ test error with $Ω (1 / ϵ^{4})$ samples and $Ω (1 / ϵ^{14})$ hidden units. Cao and Gu (2019b) consider SGD with an online oracle and give a general result. Under the same assumption as in (Cao and Gu, 2019a), their result requires $Ω (1 / ϵ^{14})$ hidden units and sample complexity $˜ O (1 / ϵ^{2})$ . By contrast, with the same online oracle, our result only needs polylogarithmic hidden units and sample complexity $˜ O (1 / ϵ)$ .

1.2 Notation

The dataset is denoted by ${(x_{i}, y_{i})}_{i = 1}^{n}$ where $xi∈\Rd$ and $yi∈\cbr−1,+1$ . For simplicity, we assume that $∥ x_{i} ∥_{2} = 1$ for any $1 \leq i \leq n$ , which is standard in the NTK literature.

The two-layer network has weight matrices $W∈\Rm×d$ and $a∈\Rm$ . We use the following parameterization, which is also used in (Du et al., 2018b; Arora et al., 2019):

f(x;W,a):=1√mm∑s=1asσ\del\ipwsx,

with initialization

ws,0∼N(0,Id),andas∼unif\del\cbr−1,+1.

Note that in this paper, $w_{s, t}$ denotes the $s$ -th row of $W$ at step $t$ . We fix $a$ and only train $W$ , as in (Li and Liang, 2018; Du et al., 2018b; Arora et al., 2019; Nitanda and Suzuki, 2019). We consider the ReLU activation $σ(z):=max\cbr0,z$ , though our analysis can be extended easily to Lipschitz continuous, positively homogeneous activations such as leaky ReLU.

We use the logistic (binary cross entropy) loss $ℓ(z):=ln\del1+exp(−z)$ and gradient descent. For any $1 \leq i \leq n$ and any $W$ , let $f_{i} (W) := f (x_{i}; W, a)$ . The empirical risk and its gradient are given by

\hcR(W):=1nn∑i=1ℓ\delyifi(W),and\hnR(W)=1nn∑i=1ℓ′\delyifi(W)yi∇fi(W).

For any $t \geq 0$ , the gradient descent step is given by $Wt+1:=Wt−ηt\hnR(Wt)$ . Also define

f(t)i(W):=\ip∇fi(Wt)W,and\hcR(t)(W):=1nn∑i=1ℓ\delyif(t)i(W).

Note that $f_{i}^{(t)} (W_{t}) = f_{i} (W_{t})$ . This property generally holds due to homogeneity: for any $W$ and any $1 \leq s \leq m$ ,

∂fi∂ws=1√mas\1\sbr\ipwsxi>0xi,and\ip∂fi∂wsws=1√masσ\del\ipwsxi,

and thus $\ip∇fi(W)W=fi(W)$ .

2 Empirical risk minimization

In this section, we consider a fixed training set and empirical risk minimization. We first state our assumption on the separability of the neural tangent model, and then give our main result and a proof sketch.

Here is some additional notation. Let $μ_{N}$ denote the Gaussian measure on $R^{d}$ , given by the Gaussian density with respect to the Lebesgue measure on $R^{d}$ . We consider the following Hilbert space

For each $1 \leq i \leq n$ , define $ϕ_{i} \in H$ by

ϕi(z):=xi\1\sbr\ipzxi>0.

One can verify that $\enVertϕi2H=1/2$ , and thus $ϕ_{i}$ is indeed in $H$ .

There exist $\barv∈H$ and $γ > 0$ such that $\enVert\barv(z)2≤1$ for any $z \in R^{d}$ , and for any $1 \leq i \leq n$ ,

yi\ip\barvϕi\cH:=yi∫\ip\barv(z)ϕi(z)\difμ\cN(z)≥γ.

A more natural assumption is that the infinite-width limit of the NTK can separate the training set with a positive margin. This assumption actually implies Section 2 (cf. Section 5). Some other discussion on the separability assumption is also given in Section 5.

With Section 2, we state our main empirical risk result. Under Section 2, given any risk target $ϵ \in (0, 1)$ and any $δ \in (0, 1 / 3)$ , let

λ:=√2ln(4n/δ)+ln(4/ϵ)γ/3,andM:=max\cbr162ln(n/δ)γ2,25ln\del2nδ,324λ2γ3.

Then for any $m \geq M$ and any constant step size $η \leq 1$ , with probability $1 - 3 δ$ over the random initialization,

1T∑t<T\hcR(Wt)≤ϵ,whereT:=⌈\nicefrac2λ2ηϵ⌉.

Moreover for any $0 \leq t < T$ ,

\enVertWt−W0F≤√6λ,and% \enVertws,t−ws,02≤3√6λγ√m for any 1≤s≤m.

While the number of hidden units required by prior work all have a polynomial dependency on $n$ , $1 / δ$ or $1 / ϵ$ , Section 2 only requires $m=Ω\delln(n/δ)+ln(1/ϵ)2$ . In the rest of Section 2, we give a proof sketch of Section 2.

2.1 Properties at initialization

In this subsection, we give some nice properties of random initialization. The proofs are given in Appendix A.

Given an initialization $(W_{0}, a)$ , for any $1 \leq s \leq m$ , define

\barus:=1√mas\barv(ws,0),

(1)

where $\barv$ is given by Section 2. Collect $\barus$ into a matrix $¯ ¯¯ ¯ U \in R^{m \times d}$ . It holds that $\enVert\barus2≤1/√m$ , and $\enVert[1]¯¯¯¯UF≤1$ . Section 2.1 ensures that with high probability $¯ ¯¯ ¯ U$ has a positive margin at initialization. Under Section 2, given any $δ \in (0, 1)$ and any $ϵ_{1} \in (0, γ)$ , if $m≥\del2ln(n/δ)/ϵ21$ , then with probability $1 - δ$ , it holds simultaneously for all $1 \leq i \leq n$ that

yif(0)i\del[1]¯¯¯¯U=yi\ip∇fi(W0)¯¯¯¯U≥γ−√2ln(n/δ)m≥γ−ϵ1.

For any $W$ , any $ϵ_{2} > 0$ , and any $1 \leq i \leq n$ , define

αi(W,ϵ2)=1mm∑s=1\1\sbr\envert\ipwsxi≤ϵ2.

Section 2.1 controls $α_{i} (W_{0}, ϵ_{2})$ . It will help us show that $¯ ¯¯ ¯ U$ has a good margin during the training process. Under the condition of Section 2.1, for any $ϵ_{2} > 0$ , with probability $1 - δ$ , it holds simultaneously for all $1 \leq i \leq n$ that

αi\delW0,ϵ2≤√2πϵ2+√ln(n/δ)2m≤ϵ2+ϵ12.

Finally, Section 2.1 controls the output of the network at initialization. Given any $δ \in (0, 1)$ , if $m \geq 25 ln (2 n / δ)$ , then with probability $1 - δ$ , it holds simultaneously for all $1 \leq i \leq n$ that

\envertf(xi;W0,a)≤√2ln\del4n/δ.

2.2 Convergence analysis of gradient descent

We analyze gradient descent in this subsection. First, define

\hcQ(W):=1nn∑i=1−ℓ′\delyifi(W).

We have the following observations.

For any $W$ and any $1 \leq s \leq m$ , $\enVert∂fi/∂ws2≤1/√m$ , and thus $\enVert∇fi(W)F≤1$ . Therefore by the triangle inequality, $\enVert\hnR(W)F≤\hcQ(W)$ .
If $ℓ$ is $1$ -Lipschitz continuous (as with the logistic loss), then $\hcQ(W)≤1$ .
If $- ℓ^{'} \leq ℓ$ (as with the logistic loss), then $\hcQ(W)≤\hcR(W)$ .

With the above observations, we give the following general result which does not require staying close to initialization. It plays an important role in the proof of our main result, and could be useful when analyzing neural networks beyond the NTK setting. For any

t \geq 0

and any

¯ ¯¯¯¯ ¯ W

, if

η_{t} \leq 1

, then

ηt\hcR(Wt)≤\enVertWt−¯¯¯¯¯¯W2F−\enVertWt+1−¯¯¯¯¯¯W2F+2ηt\hcR(t)\del[1]¯¯¯¯¯¯W.

Consequently, if we use a constant step size $η \leq 1$ for $0 \leq τ < t$ , then

η\del∑τ<t\hcR(Wτ)+\enVertWt−¯¯¯¯¯¯W2F≤\enVertW0−¯¯¯¯¯¯W2F+2η\del∑τ<t\hcR(τ)\del[1]¯¯¯¯¯¯W.

Proof.

We have

\enVertWt+1−¯¯¯¯¯¯W2F=\enVertWt−¯¯¯¯¯¯W2F−2ηt\ip\hnR(Wt)Wt−¯¯¯¯¯¯W+η2t\enVert\hnR(Wt)2F.

(2)

The first order term of eq. 2 can be handled using the convexity of $ℓ$ and homogeneity of ReLU:

\ip\hnR(Wt)Wt−¯¯¯¯¯¯W=1nn∑i=1ℓ′\delyifi(Wt)yi\ip\nfi(Wt)Wt−¯¯¯¯¯¯W=1nn∑i=1ℓ′\delyifi(Wt)\delyifi(Wt)−yif(t)i\del[1]¯¯¯¯¯¯W≥1nn∑i=1\delℓ\delyifi(Wt)−ℓ\delyif(t)i\del[1]¯¯¯¯¯¯W=\hcR(Wt)−\hcR(t)\del[1]¯¯¯¯¯¯W.

(3)

The second-order term of eq. 2 can be bounded as follows

η2t\enVert\hnR(Wt)2F≤η2t\hcQ(Wt)2≤ηt\hcQ(Wt)≤ηt\hcR(Wt),

(4)

because $\enVert\hnR(Wt)F≤\hcQ(Wt)$ , and $ηt,\hcQ(Wt)≤1$ , and $\hcQ(Wt)≤\hcR(Wt)$ . Combining eqs. 4, 3 and 2 gives

ηt\hcR(Wt)≤\enVertWt−¯¯¯¯¯¯W2F−\enVertWt+1−¯¯¯¯¯¯W2F+2ηt\hcR(t)\del[1]¯¯¯¯¯¯W.

Telescoping gives the other claim. ∎

Using Sections 2.2, 2.1, 2.1 and 2.1, we can prove Section 2. Below is a proof sketch; the full proof is given in Appendix A.

We first show that $¯ ¯¯ ¯ U$ defined in eq. 1 gives a positive margin at step $t$ as long as the activation patterns do not change too much from the initialization.
We then show that such a phase lasts for a long time with a mild overparameterization by giving a strong control of $\enVertWt−W0F$ via Section 2.2. Prior work only shows an $O (t)$ or $O (\sqrt{t})$ upper bound on $\enVertWt−W0F$ , which then requires the number of hidden units to be $\poly(1/ϵ)$ . By contrast, we are able to control $\enVertWt−W0F$ by $O (ln (t))$ , which allows us to have a $polylog (n, 1 / δ, 1 / ϵ)$ overparameterization.
Next we use Section 2.2 once again to get the empirical risk guarantee.
We also give an upper bound on $\enVertws,t−ws,02$ , or $\enVertW⊤t−W⊤02,∞$ . This will give us a Rademacher complexity bound in Section 3.

3 Generalization

To get a generalization bound, we naturally extend Section 2 to the following assumption, which is also made in (Nitanda and Suzuki, 2019) for smooth activations. There exist $\barv∈H$ and $γ > 0$ such that $\enVert\barv(z)2≤1$ for any $z \in R^{d}$ , and

y∫\ip\barv(z)x\1\sbr\ipzx>0\difμN(z)≥γ

for any $(x, y)$ sampled from the data distribution $\cD$ (i.e., almost surely over $\cD$ ).

Here is our test error bound with Section 3. Under Section 3, given any $ϵ \in (0, 1)$ and any $δ \in (0, 1 / 4)$ , let $λ$ and $M$ be given as in Section 2:

λ:=√2ln(4n/δ)+ln(4/ϵ)γ/3,andM:=max\cbr162ln(n/δ)γ2,25ln\del2nδ,324λ2γ3.

Then for any $m \geq M$ and any constant step size $η \leq 1$ , with probability $1 - 4 δ$ over the random initialization and data sampling,

P(x,y)∼\cD\delyf(x;Wk,a)≤0≤2ϵ+24\del√2ln(4n/δ)+ln(4/ϵ)γ2√n+6√ln(2/δ)2n,

where $k$ denotes the step with the minimum empirical risk before $⌈\nicefrac2λ2ηϵ⌉$ .

Below is a direct corollary of Section 3. Under Section 3, given any $ϵ, δ \in (0, 1)$ , using a constant step size and let

n=˜Ω\del1γ4ϵ2,andm=Ω\delln(n/δ)+ln(1/ϵ)2γ5,

it holds with probability $1 - δ$ that $P(x,y)∼\cD\delyf(x;Wk,a)≤0≤ϵ$ , where $k$ denotes the step with the minimum empirical risk in the first $˜O(\nicefrac1γ2ϵ)$ steps.

To prove Section 3, we consider the sigmoid mapping $- ℓ^{'} (z) = e^{- z} / (1 + e^{- z})$ , the empirical average $\hcQ(Wk)$ , and the corresponding population average $\cQ(Wk):=E(x,y)∼\cD\sbr−ℓ′\delyf(x;Wk,a)$ . First of all, since $P(x,y)∼\cD\delyf(x;Wk,a)≤0≤2\cQ(Wk)$ , it is enough to control $\cQ(Wk)$ . Next, as $\hcQ(Wk)$ is controlled by Section 2, it is enough to control the generalization error $\cQ(Wk)−\hcQ(Wk)$ . Moreover, since $- ℓ^{'}$ is supported on $[0, 1]$ and $1$ -Lipschitz, it is enough to bound the Rademacher complexity of the function space explored by gradient descent. Invoking the bound on $\enVertW⊤k−W⊤02,∞$ from Section 2 finishes the proof. The proof details are given in Appendix B.

To get Section 3, we use a Lipschitz-based Rademacher complexity bound. One can also use a smoothness-based Rademacher complexity bound (Srebro et al., 2010, Theorem 1) and get a sample complexity $˜Ω(\nicefrac1γ4ϵ)$ . However, the bound will become complicated and some large constant will be introduced. It is an interesting open question to give a clean analysis based on smoothness.

4 Stochastic gradient descent

There are some different formulations of SGD. In this section, we consider SGD with an online oracle. We randomly sample $W_{0}$ and $a$ , and fix $a$ during training. At step $i$ , a data example $(x_{i}, y_{i})$ is sampled from the data distribution. We still let $f_{i} (W) := f (x_{i}; W, a)$ , and perform the following update

Wi+1:=Wi−ηiℓ′\delyifi(Wi)yi\nfi(Wi).

Note that here $i$ starts from $0$ .

Still with Section 3, we show the following result. Under Section 3, given any $ϵ, δ \in (0, 1)$ , using a constant step size and $m=Ω\del\nicefrac\delln(1/δ)+ln(1/ϵ)2γ5$ , it holds with probability $1 - δ$ that

1nn∑i=1P(x,y)∼\cD\delyf(x;Wi,a)≤0≤ϵ,forn=˜O(\nicefrac1γ2ϵ).

Below is a proof sketch; the details are given in Appendix C. For any $W$ , define

\cRi(W):=ℓ\delyi\ip\nfi(Wi)W,% and\cQi(W):=−ℓ′\delyi\ip\nfi(Wi)W.

Due to homogeneity, $\cRi(Wi)=ℓ\delyifi(Wi)$ and $\cQi(Wi)=−ℓ′\delyifi(Wi)$ .

The first step is an extension of Section 2.2 to the SGD setting. The proofs are similar. With a constant step size $η \leq 1$ , for any $¯ ¯¯¯¯ ¯ W$ and any $i \geq 0$ ,

η\del∑t<i\cRt(Wt)+\enVertWi−¯¯¯¯¯¯W2F≤\enVertW0−¯¯¯¯¯¯W2F+2η\del∑t<i\cRt\del[1]¯¯¯¯¯¯W.

With Section 4, we can also extend Section 2 to the SGD setting and get a bound on $∑i<n\cQi(Wi)$ , using a similar proof. To further get a bound on the cumulative population risk $∑i<n\cQ(Wi)$ , the key observation is that $∑i<n\del\cQ(Wi)−\cQi(Wi)$ is a martingale. Using a martingale Bernstein bound, we prove the following lemma; applying it finishes the proof of Section 4. Given any $δ \in (0, 1)$ , with probability $1 - δ$ ,

∑t<i\cQ(Wt)≤4∑t<i\cQt(Wt)+4ln\del1δ.

5 On separability

Given a training set $\cbr(xi,yi)ni=1$ , the linear kernel is defined as $K0(xi,xj):=\ipxixj$

. The maximum margin achievable by a linear classifier is given by

γ0:=minq∈Δn√\delq⊙y⊤K0\delq⊙y,

(5)

where $(q⊙y)∈\Rn$ denotes the element-wise product of $q$ and $y$ . If the data is not linearly separable, $γ_{0} = 0$ .

In this paper we train the first layer of a two-layer network, and the kernel we consider is the NTK of the first layer:

	$K1\delxi,xj$	$:=E\sbr∂f(xi;W0,a)∂W0,∂f(xj;W0,a)∂W0$
		$=\ipxixjEw∼N(0,Id)\sbr\1\sbr\ipxiw>0\1\sbr\ipxjw>0.$

Similar to the definition of $γ_{0}$ , the margin given by $K_{1}$ is defined as

γ1:=minq∈Δn√\delq⊙y⊤K1\delq⊙y.

Regarding the relation between $γ_{1}$ and Section 2, we have the following result. If $γ_{1} > 0$ , then there exists $^v \in H$ s.t. $\enVert^vH=γ1$ , and $\enVert^v(z)2≤1$ for any $z \in R^{d}$ , and $yi\ip^vϕi\cH≥γ21$ for any $1 \leq i \leq n$ . The proof is given in Appendix D, and uses Fenchel duality theory. $^v$ given by Section 5 satisfies Section 2 with $γ = γ_{1}^{2}$ , but there might exist some $\barv$ with a much better $γ$ , since the upper bound $\enVert^v(z)2≤1$ in Section 5 might be very loose.

We can further ask how large $γ_{1}$ could be. Oymak and Soltanolkotabi (2019, Corollary I.2)

show that if for any two feature vectors

x_{i}

and

x_{j}

, we have

∥ x_{i} - x_{j} ∥_{2} \geq θ

and

∥ x_{i} + x_{j} ∥_{2} \geq θ

for some

θ > 0

, then

λ_{0} := λ_{min} (K_{1}) \geq \frac{θ}{100 n^{2}} .

For arbitrary labels $y \in {- 1, + 1}^{n}$ , since $\enVertq⊙y2≥1/√n$ , we have the worst case bound $γ21≥\nicefracθ100n3$ . However, real world labels could give a much better $γ_{1}$ . For example, a tighter lower bound on $γ_{1}$ is $\nicefracθ100n3S$ , where $n_{S}$ denotes the number of support vectors, which might be much smaller than $n$ .

On the other hand, given any training set $\cbr(xi,yi)ni=1$ which may have a large margin, if we replace $y$ with random labels $ϵ∼unif\del{−1,+1}n$ , with high probability the margin becomes $O (1 / \sqrt{n})$ . To see this, let $^q$ denote the uniform probability vector $(\nicefrac1n,…,\nicefrac1n)$ . Note that

	$\bbEϵ∼unif\del{−1,+1}n\sbr\del^q⊙ϵ⊤K1\del^q⊙ϵ$	$=\bbEϵ∼unif\del{−1,+1}n\sbrn∑i,j=11n2ϵiϵjK1(xi,xj)$
		$=1n2n∑i,j=1\bbEϵ∼unif\del{−1,+1}n\sbrϵiϵjK1(xi,xj)$
		$= \frac{1}{n^{2}} n \sum i = 1 K_{1} (x_{i}, x_{i}) = \frac{1}{2 n} .$

Since $0≤\del^q⊙ϵ⊤K1\del^q⊙ϵ≤1$ for any $ϵ$ , by Hoeffding’s inequality it holds with high probability that $\del^q⊙ϵ⊤K1\del^q⊙ϵ=O(1/n)$ , and thus the margin is $O (1 / \sqrt{n})$ .

6 Open problems

In this paper, we analyze gradient descent on a two-layer network in the NTK regime, where the weights stay close to the initialization. It is an interesting open question if gradient descent learns something beyond the NTK, after the iterates move far enough from the initial weights. It is also interesting to extend our analysis to other architectures, such as multi-layer networks, convolutional networks, and residual networks. Finally, in this paper we only discuss binary classification; it is interesting to see if it is possible to get similar results for other tasks, such as regression.

References

Allen-Zhu et al. (2018a) Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. arXiv preprint arXiv:1811.04918, 2018a.
Allen-Zhu et al. (2018b) Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameterization. arXiv preprint arXiv:1811.03962, 2018b.
Arora et al. (2019) Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. arXiv preprint arXiv:1901.08584, 2019.
Bartlett and Mendelson (2002) Peter L. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. JMLR, 3:463–482, Nov 2002.
Beygelzimer et al. (2011) Alina Beygelzimer, John Langford, Lihong Li, Lev Reyzin, and Robert Schapire.
Contextual bandit algorithms with supervised learning guarantees.
In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics
, pages 19–26, 2011.
Borwein and Zhu (2005) Jonathan M. Borwein and Qiji J. Zhu. Techniques of Variational Analysis, volume 20 of. CMS Books in Mathematics, 2005.
Cao and Gu (2019a) Yuan Cao and Quanquan Gu. Generalization error bounds of gradient descent for learning over-parameterized deep relu networks. arXiv preprint arXiv:1902.01384, 2019a.
Cao and Gu (2019b) Yuan Cao and Quanquan Gu. Generalization bounds of stochastic gradient descent for wide and deep neural networks. arXiv preprint arXiv:1905.13210, 2019b.
Chizat and Bach (2019) Lenaic Chizat and Francis Bach. A Note on Lazy Training in Supervised Differentiable Programming. arXiv:1812.07956v2 [math.OC], 2019.
Du et al. (2018a) Simon S Du, Jason D Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. arXiv preprint arXiv:1811.03804, 2018a.
Du et al. (2018b) Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint arXiv:1810.02054, 2018b.
Jacot et al. (2018) Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in neural information processing systems, pages 8571–8580, 2018.
Li and Liang (2018) Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data. In Advances in Neural Information Processing Systems, pages 8157–8166, 2018.
Liang (2016) Percy Liang.
Stanford CS229T/STAT231: Statistical Learning Theory, Apr 2016.
URL https://web.stanford.edu/class/cs229t/notes.pdf.
Nitanda and Suzuki (2019) Atsushi Nitanda and Taiji Suzuki. Refined generalization analysis of gradient descent for over-parameterized two-layer neural networks with smooth activations on classification problems. arXiv preprint arXiv:1905.09870, 2019.
Oymak and Soltanolkotabi (2019) Samet Oymak and Mahdi Soltanolkotabi. Towards moderate overparameterization: global convergence guarantees for training shallow neural networks. arXiv preprint arXiv:1902.04674, 2019.
Shalev-Shwartz and Ben-David (2014) Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms
. Cambridge University Press, 2014.
Srebro et al. (2010) Nathan Srebro, Karthik Sridharan, and Ambuj Tewari. Smoothness, low noise and fast rates. In Advances in neural information processing systems, pages 2199–2207, 2010.
Wainwright (2015) Martin J. Wainwright. UC Berkeley Statistics 210B, Lecture Notes: Basic tail and concentration bounds, Jan 2015. URL https://www.stat.berkeley.edu/~mjwain/stat210b/Chap2_TailBounds_Jan22_2015.pdf.
Zou and Gu (2019) Difan Zou and Quanquan Gu. An improved analysis of training over-parameterized deep neural networks. arXiv preprint arXiv:1906.04688, 2019.
Zou et al. (2018) Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Stochastic gradient descent optimizes over-parameterized deep relu networks. arXiv preprint arXiv:1811.08888, 2018.

Appendix A Omitted proofs from Section 2

Proof of Section 2.1.

By Section 2, given any $1 \leq i \leq n$ ,

μ:=Ew∼N(0,Id)\sbryi\ip\barv(w)xi\1\sbr\ipwxi>0≥γ.

On the other hand,

yif(0)i\del[1]¯¯¯¯U=1mm∑s=1yi\ip\barv(ws,0)xi\1\sbr\ipws,0xi>0

is the empirical mean of i.i.d. r.v.’s supported on $[- 1, + 1]$ with mean $μ$ . Therefore by Hoeffding’s inequality, with probability $1−\nicefracδn$ ,

yif(0)i\del[1]¯¯¯¯U−γ≥yif(0)i\del[1]¯¯¯¯U−μ≥−√2ln(n/δ)m.

Applying a union bound finishes the proof. ∎

Proof of Section 2.1.

Given any fixed $ϵ_{2}$ and $1 \leq i \leq n$ ,

E\sbrαi(W0,ϵ2)=P\del\envert\ipwxi≤ϵ2≤2ϵ2√2π=√2πϵ2,

because $\ipwxi$ is a standard Gaussian r.v. and the density of standard Gaussian has maximum $1 / \sqrt{2 π}$ . Since $α_{i} (W_{0}, ϵ_{2})$ is the empirical mean of Bernoulli r.v.’s, by Hoeffding’s inequality, with probability $1−\nicefracδn$ ,

αi(W0,ϵ2)≤E\sbrαi(W0,ϵ2)+√ln(n/δ)2m≤√2πϵ2+√ln(n/δ)2m.

Applying a union bound finishes the proof. ∎

To prove Section 2.1, we need the following technical result.

Consider the random vector $X = (X_{1}, \dots, X_{m})$ , where $X_{i} = σ (Z_{i})$ for some $σ : R \to R$ that is $1$ -Lipschitz, and $Z_{i}$ are i.i.d. standard Gaussian r.v.’s. Then the r.v. $∥ X ∥_{2}$ is $1$ -sub-Gaussian, and thus with probability $1 - δ$ ,

∥X∥2−E\sbr∥X∥2≤√2ln(1/δ).

Proof.

Given $a \in R^{m}$ , define

f(a)= ⎷m∑i=1σ(ai)2=\enVertσ(a)2,

where $σ (a)$ is obtained by applying $σ$ coordinate-wisely to $a$ . For any $a, b \in R^{m}$ , by the triangle inequality, we have

\envertf(a)−f(b)=\envert\enVertσ(a)2−\enVertσ(b)2

≤\enVertσ(a)−σ(b)2= ⎷m∑i=1\delσ(ai)−σ(bi)2,

and by further using the $1$ -Lipschitz continuity of $σ$ , we have

\envertf(a)−f(b)≤ ⎷m∑i=1\delσ(ai)−σ(bi)2≤ ⎷m∑i=1(ai−bi)2=\enVerta−b2.

As a result, $f$ is a $1$ -Lipschitz continuous function w.r.t. the $ℓ_{2}$ norm, indeed $f (X)$ is $1$ -sub-Gaussian and the bound follows by Gaussian concentration (Wainwright, 2015, Theorem 2.4). ∎

Proof of Section 2.1.

Given $1 \leq i \leq n$ , let $h_{i} = σ (W_{0} x_{i}) / \sqrt{m}$ . By Appendix A, $∥ h_{i} ∥_{2}$

is sub-Gaussian with variance proxy

1 / m

, and with probability at least

1−\nicefracδ2n

over

W_{0}

∥hi∥2−E\sbr∥hi∥2≤√2ln(2n/δ)m≤√2ln(2n/δ)25ln(2n/δ)≤1−√22.

On the other hand, by Jensen’s inequality,

E\sbr∥hi∥2≤√E\sbr∥hi∥22=√22.

As a result, with probability $1−\nicefracδ2n$ , it holds that $∥ h_{i} ∥_{2} \leq 1$ . By a union bound, with probability $1−\nicefracδ2$ over $W_{0}$ , for all $1 \leq i \leq n$ , we have $∥ h_{i} ∥_{2} \leq 1$ .

For any $W_{0}$ such that the above event holds, and for any $1 \leq i \leq n$ , the r.v. $\iphia$ is sub-Gaussian with variance proxy $∥ h_{i} ∥_{2}^{2} \leq 1$ . By Hoeffding’s inequality, with probability $1−\nicefracδ2n$ over $a$ ,

\envert\iphia=\envertf(xi;W0,a)≤√2ln\del4n/δ.

By the union bound, with probability $1−\nicefracδ2$ over $a$ , for all $1 \leq i \leq n$ , we have $\envertf(xi;W0,a)≤√2ln\del4n/δ$ .

The probability that the above events all happen is at least $(1−\nicefracδ2)(1−\nicefracδ2)≥1−δ$ , over $W_{0}$ and $a$ . ∎

Proof of Section 2.

The condition on $m$ ensures that Sections 2.1, 2.1 and 2.1 hold with $ϵ_{1} = γ / 9$ and $ϵ_{2} = γ / 3$ .

For any $1 \leq i \leq n$ and any step $t$ , let $ξ_{i, t}$ denote the proportion of activation patterns for $x_{i}$ that are different from step $0$ to step $t$ . Formally,

ξi,t:=1mm∑s=1\1

Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks

Random Feature Amplification: Feature Learning and Generalization in Neural Networks

Quantifying the Benefit of Using Differentiable Learning over Tangent Kernels

Feature selection with gradient descent on two-layer networks in low-rotation regimes

On the Implicit Biases of Architecture Gradient Descent

A Study of Neural Training with Non-Gradient and Noise Assisted Gradient Methods

Nonparametric Regression with Shallow Overparameterized Neural Networks Trained by GD with Early Stopping

Preconditioned Gradient Descent Algorithm for Inverse Filtering on Spatially Distributed Networks

1 Introduction

Our contributions.

1.1 Related work

1.2 Notation

2 Empirical risk minimization

2.1 Properties at initialization

2.2 Convergence analysis of gradient descent

Proof.

3 Generalization

4 Stochastic gradient descent

5 On separability

6 Open problems

References

Appendix A Omitted proofs from Section 2

Proof of Section 2.1.

Proof of Section 2.1.

Proof.

Proof of Section 2.1.

Proof of Section 2.

Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks

Related Research

Random Feature Amplification: Feature Learning and Generalization in Neural Networks

Quantifying the Benefit of Using Differentiable Learning over Tangent Kernels

Feature selection with gradient descent on two-layer networks in low-rotation regimes

On the Implicit Biases of Architecture Gradient Descent

A Study of Neural Training with Non-Gradient and Noise Assisted Gradient Methods

Nonparametric Regression with Shallow Overparameterized Neural Networks Trained by GD with Early Stopping

Preconditioned Gradient Descent Algorithm for Inverse Filtering on Spatially Distributed Networks

1 Introduction

Our contributions.

1.1 Related work

1.2 Notation

2 Empirical risk minimization

2.1 Properties at initialization

2.2 Convergence analysis of gradient descent

Proof.

3 Generalization

4 Stochastic gradient descent

5 On separability

6 Open problems

References

Appendix A Omitted proofs from Section 2

Proof of Section 2.1.

Proof of Section 2.1.

Proof.

Proof of Section 2.1.

Proof of Section 2.