On the convergence of gradient descent for two layer neural networks

1 Introduction

The universal approximation theorem tells us that a two-layer neural networks can approximate a broad class of functions, provided that the width is sufficiently large, i.e. in the over-parametrized regime [1, 2]. Recently, it has been shown that gradient descent can find the optimal parameters for the wide two layer neural networks using optimal transport theory [3, 4, 5]

, and using direct estimation on particular neural network structure

[6, 7, 8]

. All these results are in the over-parametrized regime. We are interested to see whether the loss function can converge to zero in the under-parametrized regime, i.e., the number of data points are larger than the width of the two layer neural networks.

Following [4, 6], we consider the loss function given by the quadratic function

ℓ (x, w) = \frac{1}{2} \int_{D} | f^{(n)} (x, w) - f (x) |^{2} d ν (x),

(1.1)

where $D \subset R^{d}$ is a compact set and $f \in C (D)$ , $d$ is the dimension, and $ν$ is some probability measure on $D$ . We aim to show that under suitable setup, with high probability, the convergence rate is exponentially fast, and we desire the width of the neural networks is independent of the training data, by using suitable approximation $f^{(n)}$ . Note that this does not contradict with the result of [9], which shows that the under-parametrized networks cannot approximate the function well. The reason is that the loss function here only measures the approximation at the given data points, instead of approximation for the whole function.

Here, we mention the two main works that motivate this work. In [4], the following form of approximations were considered inspired by weak approximation of measures

{~ f}^{(n)} (x, t) := \frac{1}{n} n \sum i = 1 a_{i} (t) φ (x, w_{i} (t)),

(1.2)

where $φ (\cdot, w)$ represents any unit function. For example, it could be the Gaussian, the one-layer neural networks or deep neural networks. They were able study the dynamics as the interacting particle systems, so that the decaying of the loss function can be understood as gradient flows in Wasserstein-2 space using optimal transport theory. However, the decaying can only be shown in the limiting regime where $n \to \infty$ and the decaying rate is unknown. The mechanism is that the parameters $(a_{i}, w_{i})$ will converge to some minimization regions.

In the work [6] by S. Du et al., the two layer of neural networks were studied, i.e.,

φ (x, w) = σ (w \cdot x), w \in R^{d} .

Here, $σ$

is the so-called activation function. In their work, the following form of apprxomation is used:

f^{(n)} (x, t) := \frac{1}{\sqrt{n}} n \sum i = 1 a_{i} (t) φ (x, w_{i} (t)) = \frac{1}{\sqrt{n}} n \sum i = 1 a_{i} (t) σ (w_{i} (t) \cdot x) .

(1.3)

Though this seems to be a little change, the dynamics has been changed significantly compared with (1.2). They are actually able to show that the weights $w_{i}$ ’s will not converge under the gradient descent method: they are close to the initialization. They claim that the loss function converges to zero exponentially in the overparamterized regime. They obtain this result by considering the dynamics of the prediction $u_{i} (t) := f^{(n)} (x_{i}, t)$ . In [6], the convergence largely is due to their Assumption 3.1 and Theorem 3.1. The analysis for their $H^{\infty}$

matrix needs to exclude the cases that there are two vectors being nearly parallel, which is not desired in the

M \to \infty

regime, where

M

is the number of training data. We note that if we rely on the dynamics of

a

, the matrix can be positive definite simply due to the positivity of

σ

, and this is the strategy we adopt here.

Below, we aim to combine and improve the ideas from these two works to show a convergence rate of the gradient descent independent of the sample size $M$ . We use the approximation from [6], i.e. (1.3), to study the dynamics of the training loss $ℓ$ directly. We makes use the key observation from [6] that the weights are close to the initial values, but instead rely on the dynamics on $a_{i} (t)$ ’s to obtain the positive definiteness of the Gram matrix. Another important technique is the generic chaining [10] from probability theory that kind of guarantees the the Gram matrix is close to some positive definite matrix uniformly data $x, y \in D$ . In section 2, we set up the mathematical formulation and state the main results. The proof is then performed in section 3. Lastly, we make some discussion in section 4.

2 Mathematical setup and the main result

We focus on the following special setup, and consider two layer neural networks as in (1.3). The generalization to other cases will be done soon in subsequent works.

About the data set and measure for $x$ , we assume the following:

Assumption 1.

The domain $D \subset R^{d}$ is a compact set such that $0 < {inf}_{x \in D} | x | \leq {sup}_{x \in D} | x | < \infty$ . The measure $ν$ is a probability measure supported on a countable subset of $D$ .

Below, we will denote the support of $ν$ by $T$ :

T := supp D .

(2.1)

One common example of $ν$ is the empirical measure

ν := \frac{1}{M} M \sum p = 1 δ (\cdot - x_{p}),

(2.2)

where ${x_{p}}$ ’s are the training data sampled i.i.d from some underlying distribution. Of course, in our discussion in this work, ${x_{p}}$ ’s are fixed.

Moreover, we assume the initial conditions as follows.

Assumption 2.

For the weight $w_{i} = (w_{i}^{k})_{k = 1}^{d}$ , we assume

w_{i} (0) \sim N (0, I_{d}), i . i . d .,

and $a_{i} (0)$ ’s are i.i.d such that $P (a_{i} (0) = 1) = P (a_{i} (0) = - 1) = \frac{1}{2}$ .

The condition $E a_{i} (0) = 0$ is important because it guarantees that the approximation (1.3) is $O (1)$ with high probability at $t = 0$ to close up the estimation later (Lemma 4). The distribution for $w_{i} (0)$

’s are assumed to be Gaussians only for technical convenience. The argument can be extended to other distributions with finite second moments.

Moreover, we assume

Assumption 3.

The activation function satisfies $σ \geq 0$ , and $σ (0) > 0$ . Moreover, $σ \in C_{b}^{1}$ , and thus

∥ σ ∥_{\infty} + ∥ σ^{'} ∥_{\infty} < \infty .

The sigmoid functions should satisfy the condition. This condition may be too strong, however, we impose this only for simplicity. The generalization to other functions should be doable.

Under the gradient descent with loss function (1.1), the parameters $w_{i}$ ’s satisfy

{˙ w}_{i} (t) = - \nabla_{w_{i}} ℓ = - \frac{1}{\sqrt{n}} \int_{D} (f^{(n)} (x) - f (x)) a_{i} (t) σ^{'} (w_{i} \cdot x) x d ν (x),

(2.3)

where ${˙ w}_{i}$ means the time derivative. Similarly, the parameters $a_{i}$ ’s satisfy

{˙ a}_{i} (t) = - \frac{\partial ℓ}{\partial a_{i}} = - \frac{1}{\sqrt{n}} \int_{D} (f^{(n)} (x) - f (x)) σ (w_{i} \cdot x) d ν (x) .

(2.4)

Instead of studying the prediction $u_{i} = f^{(n)} (x_{i}, t)$ as in [6], we study the dynamics of the loss function directly as motivated by [4],

(2.5)

The matrix

M (x, y) := \frac{1}{n} n \sum i = 1 (σ (w_{i} (t) \cdot x) σ (w_{i} (t) \cdot y) + a_{i}^{2} (t) σ^{'} (w_{i} (t) \cdot x) σ^{'} (w_{i} (t) \cdot y) x \cdot y)

(2.6)

is called the Gram matrix in the terminology from [6]. We aim to use the positive definiteness of this matrix to show the exponential convergence. The second part is positive semi-definite (in fact in [6], it was argued that this part is also positive definite when the data points are not parallel), so we can control $\frac{d}{d t} ℓ$ easily as

\frac{d}{d t} ℓ \leq - \frac{1}{n} n \sum i = 1 \iint_{D \times D} (f^{(n)} (x) - f (x)) (f^{(n)} (y) - f (y)) σ (w_{i} (t) \cdot x) σ (w_{i} (t) \cdot y) d ν (x) d ν (y) .

(2.7)

One key observation from [6] is that the weights do not change much during the dynamics. The choice (1.3) ensures the form $\frac{1}{n} \sum_{i = 1}^{n}$

, so that one has kind of law of large number convergence here. Moreover, we note that

{(w_{i} (0) \cdot x, w_{i} (0) \cdot y)}_{i = 1}^{n}

are i.i.d

2

D Gaussian variables regardless the dimension of

x

itself. Using the generic chaining approach [10], this allows us obtain the convergence rate of the loss function independent of

M

and

d

when

n

is large enough (

n \geq d C | log δ |^{2}

without exponential dependence in

d

Theorem 1.

Under Assumptions 1-3, for any $δ > 0$ small, when $n \geq C (d + \sqrt{∥ f ∥_{\infty}}) (log δ)^{2}$ , with probability $1 - δ$ , it holds that

ℓ (t) \leq C log (\frac{1}{δ}) exp (- \frac{λ}{2} t),

(2.8)

where $C$ and $λ$ are independent of $n, ν$ , and $d$ .

We remark that as $ℓ (t) \to 0$ does not mean $f^{(n)} (\cdot)$ approximates $f (\cdot)$ well. It only approximates the function values on the discrete points ${x_{p}}_{p = 1}^{M}$ . However, since the convergence is independent of $M$ , this result somehow says that any continuous function can be approximated by the two-layer neural networks with high probability.

In fact, it seems that the following approximation ability of two layer neural networks holds.

Corollary 1.

Suppose $D$ satisfying Assumptions 1 is the union of closures of some domains (open connected sets) so that the Lebesgue measure can be defined. Then, the two layer neural networks satisfying Assumptions 3 with width $n ≳ d$ are able to approximate any functions $f \in C (D)$ , $∥ f ∥_{\infty} \leq 1$ , under the $L^{2} (d x)$ norm.

Here, $L^{2} (d x)$ means the $L^{2}$ norm with Lebesgue measure,. As commented in section 4, it seems that such good approximations are available ”everywhere” in the parameter space.

3 The proof of the main result

Below, we use $C$ to indicate a constant independent of $ν$ $d, n$ , whose concrete meaning can change from line to line.

Lemma 1.

The derivatives of the parameters satisfy

\begin{matrix} \frac{1}{2} \frac{d}{d t} n \sum i = 1 a_{i}^{2} & = - \int_{D} (f^{(n)} (x) - f (x)) f^{(n)} (x) d ν (x) \leq ∥ f ∥_{\infty} \int_{D} | f^{(n)} (x) - f (x) | d ν (x) \leq ∥ f ∥_{\infty} \sqrt{ℓ (t)}, \end{matrix}

(3.1)

and

| {˙ w}_{i} | \leq C \frac{| a_{i} (t) |}{\sqrt{n}} \sqrt{ℓ (t)},

(3.2)

where $C$ is independent of $i, n, M$ and $d$ .

The proof is straightforward and we omit. Let us just remark that if we consider, for example, ${˙ w}_{i}$ . Using (2.3), by Hölder’s inequality, it is straightforward to obtain

\begin{matrix} | {˙ w}_{i} (t) | & \leq \frac{}{1} \sqrt{n} ∣ ∣ ∣ \int_{D} (f^{(n)} (x) - f (x)) a_{i} (t) σ^{'} (w_{i} \cdot x) x d ν (x) ∣ ∣ ∣ \leq \frac{| a_{i} (t) |}{\sqrt{n}} {(\int_{D} (f^{(n)} (x) - f (x))^{2} d ν (x))}_{D}^{1 / 2} {(\int_{D} ∥ σ^{'} ∥_{\infty}^{2} d ν (x))}_{D}^{1 / 2} \leq C \frac{| a_{i} (t) |}{\sqrt{n}} \sqrt{ℓ (t)} . \end{matrix}

Using Lemma 1, we are able to establish:

\begin{matrix} \frac{d}{d t} ℓ (t) \leq & - \frac{1}{n} n \sum i = 1 \iint_{D \times D} (f^{(n)} (x) - f (x)) (f^{(n)} (y) - f (y)) \times σ (w_{i} (0) \cdot x) σ (w_{i} (0) \cdot y) d ν (x) d ν (y) + R (t) =: & - D_{n} (t) + R (t), \end{matrix}

(3.3)

where we have introduced

D_{n} (t) := \frac{1}{n} n \sum i = 1 \iint_{D \times D} (f^{(n)} (x) - f (x)) (f^{(n)} (y) - f (y)) σ (w_{i} (0) \cdot x) σ (w_{i} (0) \cdot y) d ν (x) d ν (y),

(3.4)

and

R (t) := \frac{C}{n} n \sum i = 1 \iint_{D \times D} | f^{(n)} (x) - f (x) | | f^{(n)} (y) - f (y) | \times \int_{0}^{t} \frac{| a_{i} (s) |}{\sqrt{n}} \sqrt{ℓ (s)} d s d ν (x) d ν (y) .

(3.5)

We now estimate these two terms separately.

Lemma 2.

The term $R (t)$ satisfies

R (t) \leq C ⎛ ⎝ \frac{1}{\sqrt{n}} {(\frac{\sum_{i = 1}^{n} a_{i} (0)^{2}}{n})}^{1 / 2} b (t) + \frac{\sqrt{∥ f ∥_{\infty}}}{n} b (t)^{3 / 2} ⎞ ⎠ ℓ (t),

where

b (t) := \int_{0}^{t} \sqrt{ℓ (s)} d s .

(3.6)

Proof.

First of all, one finds

\frac{1}{n} n \sum i = 1 \frac{| a_{i} (s) |}{\sqrt{n}} \leq \frac{1}{n} {(n \sum i = 1 a_{i}^{2} (s))}^{1 / 2} .

Using (3.1), one has

n \sum i = 1 a_{i}^{2} (s) \leq n \sum i = 1 a_{i}^{2} (0) + 2 ∥ f ∥_{\infty} \int_{0}^{s} \sqrt{ℓ (τ)} d τ .

Hence, due to $\sqrt{a + b} \leq \sqrt{a} + \sqrt{b}$ for $a \geq 0, b \geq 0$ , one has

\frac{1}{n} n \sum i = 1 \frac{| a_{i} (s) |}{\sqrt{n}} \leq \frac{1}{n} {(n \sum i = 1 a_{i}^{2} (0))}^{1 / 2} + \frac{\sqrt{2 ∥ f ∥_{\infty}}}{n} {(\int_{0}^{s} \sqrt{ℓ (τ)} d τ)}^{1 / 2} .

Consequently, one has

\frac{1}{n} n \sum i = 1 \int_{0}^{t} \frac{| a_{i} (s) |}{\sqrt{n}} \sqrt{ℓ (s)} d s \leq \frac{1}{n} {(n \sum i = 1 a_{i}^{2} (0))}^{1 / 2} \int_{0}^{t} \sqrt{ℓ (s)} d s + \frac{2 \sqrt{2}}{3} \frac{\sqrt{∥ f ∥_{\infty}}}{n} {(\int_{0}^{t} \sqrt{ℓ (s)} d s)}^{3 / 2},

(3.7)

where we have used the fact that

{(\int_{0}^{s} \sqrt{ℓ (τ)} d τ)}^{1 / 2} \sqrt{ℓ (s)} = \frac{2}{} 3 \frac{d}{d s} {(\int_{0}^{s} \sqrt{ℓ (τ)} d τ)}^{3 / 2} .

The claim then follows. ∎

We are now in a position to take $n \to \infty$ in $D_{n} (t)$ . In particular, we need to estimate

D_{n} (x, y) := \frac{1}{n} n \sum i = 1 σ (w_{i} (0) \cdot x) σ (w_{i} (0) \cdot y) .

For fixed $x, y$ , it is clear that this converges to

D (x, y) := E_{w \sim N (0, I_{d})} σ (w \cdot x) σ (w \cdot y),

and the rate is independent of $d$ . However, to obtain the uniform convergence rate for $D_{n} (x, y)$ is challenging. The usual technique is the union bound, which gives a quantity involving $M^{2}$ , and this further requires the over-parametrized regime. One may also use the convergence of empirical measures in the so called Wasserstein spaces [11], which yields a quantity independent of the training data, but suffers from curse of dimensionality. In our problem here, we are essentially considering the projections of high dimensional Gaussians. Hence, we expect the convergence rate will not suffer from curse of dimensionality. We find that the generic chaining approach in [10, 12] useful.

We introduce the following result essentially derived from [12, Proposition 2.2] and the proof of [12, Theorem 1.1]. The proof is then deferred to Appendix A using generic chaining approach.

Proposition 1.

Consider

{Z_{x} := \frac{1}{n} n \sum i = 1 Y_{x}^{i}}_{x \in D} .

Suppose that for any given $x$ , ${Y_{x}^{i}}$ ’s are independent, and for any $x \neq y$ , $Y_{x}^{i}$ and $Y_{y}^{j}$ are independent for $i \neq j$ . Moreover, ${Y_{x}^{i}}$ ’s have the same distribution, satisfying the following.

They are uniformly bounded, have mean zero, and finite variance.
there is a constant $C_{1}$ such that

$E | Y_{x}^{i} - Y_{y}^{i} |^{2} \leq C_{1} | x - y |^{2}, \forall i, x, y,$

then there is a parameter $γ_{2}$ only depending on $D$ with magnitude $γ_{2} \leq C \sqrt{d}$ , such that for any small $δ > 0$ and $T \subset D$ that is countable, when $n \geq C γ_{2}^{2} (log (\frac{1}{δ}))^{2}$ , with probability $1 - δ$ ,

sup x \in T | Z_{x} | \leq C log (\frac{1}{δ}) \frac{γ_{2}}{\sqrt{n}},

(3.8)

where $C$ is independent of $d, T, n$ .

We now move to the estimates of $D_{n} (t)$ . We have

Lemma 3.

For any small $δ > 0$ , when $n \geq C γ_{2}^{2} (log (\frac{1}{δ}))^{2}$ , with probability $1 - δ$ , it holds that

sup x, y \in T | D_{n} (x, y) - D (x, y) | \leq C log (\frac{1}{δ}) \frac{γ_{2}}{\sqrt{n}} .

(3.9)

Proof.

We only need to verify the conditions in Proposition 1. Here, the domain we work on is $D \times D$ while $T \times T \subset D \times D$ is the countable subset we consider. The distance is defined by $| (x, y) - (x_{1}, y_{1}) | := \sqrt{| x - x_{1} |^{2} + | y - x_{1} |^{2}}$ .

Clearly, we need to identity

Y_{x, y}^{i} = σ (w_{i} (0) \cdot x) σ (w_{i} (0) \cdot y) - E_{w \sim N (0, I_{d})} σ (w \cdot x) σ (w \cdot y) .

Then,

| Y_{x, y}^{i} - Y_{x_{1}, y_{1}}^{i} |^{2} \leq C (| w_{i} (0) \cdot (x - x_{1}) |^{2} + | w_{i} (0) \cdot (y - y_{1}) |^{2}) .

Taking expectation, one finds $E | w_{i} (0) \cdot (x - x_{1}) |^{2} = | x - x_{1} |^{2}$ . The claim follows.

Note that in this case, the dimension is $2 d$ , but clearly, $γ_{2} (2 d) \leq C γ_{2} (d)$ . ∎

Note that the boundedness of $σ$ is also used here for the Bernstein inequality in Appendix A. Consequently, when (3.9) holds, one has

| D_{n} (t) - D (t) | \leq C log (\frac{1}{δ}) \frac{γ_{2}}{\sqrt{n}} ℓ (t) .

(3.10)

Now, we move onto the initial values of $ℓ (0)$ . If we identify

Y_{x}^{i} := a_{i} (0) σ (w_{i} (0) \cdot x)

and use Proposition 1, we can find that when $n \geq C γ_{2}^{2} (log (\frac{1}{δ}))^{2}$ , with probability $1 - δ$ , it holds that

sup x \in T ∣ ∣ \frac{1}{\sqrt{n}} n \sum i = 1 a_{i} (0) σ (w_{i} (0) \cdot x) ∣ ∣ \leq C log (\frac{1}{δ}) γ_{2} .

(3.11)

Here, $γ_{2}$ depends on $d$ . However, according to [8, Lemma 1], one has the following.

Lemma 4.

For any small $δ > 0$ , with probability $1 - δ$ , it holds that

sup x \in T ∣ ∣ \frac{1}{\sqrt{n}} n \sum i = 1 a_{i} (0) σ (w_{i} (0) \cdot x) ∣ ∣ \leq C log (\frac{1}{δ}) .

(3.12)

The proof is done via the Rademacher complexity, and we refer the readers to [8].

Now, we estimate the Gram matrix.

Lemma 5.

There exists $λ > 0$ independent of $d, n, M$ such that

E_{w \sim N (0, I_{d})} σ (w \cdot x) σ (w \cdot y) \geq λ, \forall x, y \in D .

(3.13)

Proof.

In fact, $(w \cdot x, w \cdot y) \sim (z_{1}, z_{2})$ , which is a 2D Gaussian with covariant matrix

Σ = (\begin{matrix} | x |^{2} & x \cdot y x \cdot y & | y |^{2} \end{matrix}) .

Consider the interval $I$ where ${inf}_{u \in I} σ \geq \frac{1}{2} σ (0)$ . Then, $I$ contains $(- ς, ς)$ for some $ς > 0$ .

Now, we should estimate $P ((z_{1}, z_{2}) \in I \times I)$ . The expectation can be rewritten as

where $ζ_{1}$ and $ζ_{2}$ are independent standard normal variables, and

r = \frac{x \cdot y}{| x | | y |} .

Note that $D$ is bounded, hence, we only need $| ζ_{1} | \in δ$ and $| ζ_{2} | \leq ϱ$ where $ϱ$ is a number chosen such that $ϱ | x |$ and $| y | (\sqrt{1 - r^{2}} + r) ϱ \leq \sqrt{2} | y | ϱ$ are in $(- ς, ς)$ for all $x, y$ . Clearly can be made universal independent of $x, y$ . The probability hence has a lower bound $p_{*}$ independent of $x, y$ . Then, setting $λ := \frac{1}{4} σ^{2} (0) p_{*}$ will suffice. ∎

We are now able to prove the main result.

Proof of Theorem 1.

By the previous lemmas, for any $δ > 0$ small, when $n \geq C γ_{2}^{2} (log δ)^{2}$ , we may pick an event $A$ with probability $P (A) = δ$ , such that outside this, we have $∣ ∣ ∣ \frac{\sum_{i = 1}^{n} a_{i}^{2} (0)}{n} ∣ ∣ ∣ \leq C$ , $ℓ (0) \leq C log (\frac{1}{δ})$ , and also $| D_{n} (t) - D (t) | \leq C log (\frac{1}{δ}) γ_{2} \frac{ℓ (t)}{\sqrt{n}}$ .

Then, on $A^{c}$

\frac{d ℓ (t)}{d t} \leq - λ ℓ (t) + C log (\frac{1}{δ}) \frac{}{γ_{2} ℓ (t)} \sqrt{n} + C (\frac{b (t)}{\sqrt{n}} + \frac{\sqrt{∥ f ∥_{\infty}}}{n} b (t)^{3 / 2}) ℓ (t) .

Define

T_{b} := inf {t \geq 0 : λ - C log (\frac{1}{δ}) \frac{γ_{2}}{\sqrt{n}} - \frac{C}{\sqrt{n}} b (s) - \frac{C \sqrt{∥ f ∥_{\infty}}}{n} b (s)^{3 / 2} > \frac{λ}{2}, \forall 0 \leq s \leq t} .

We claim that if $n \geq C_{1} γ_{2}^{2} (log δ)^{2}$ for some $C_{1}$ , then $T_{b} = \infty$

We first of all note that $ℓ (t)$ is nonincreasing. We now consider a fixed $ω \in A^{c}$ . Then, all the quantities below are essentially discussed for such $ω$ , and we omit this dependence for convenience.

In fact, since $b (0) = 0$ , it is then clear that if $n > 16 C^{2} γ_{2}^{2} (log δ)^{2} / λ^{2}$ , $T_{b} > 0$ . Then, for $t \in [0, T_{b})$ , then

\frac{d ℓ (t)}{d t} \leq - \frac{1}{2} λ ℓ (t) .

Then,

ℓ (t) \leq ℓ (0) exp (- \frac{λ}{2} t) \leq C log (\frac{1}{} δ) exp (- \frac{λ}{2} t) .

Hence, for all $t \in [0, T_{b})$

b (t) = \int_{0}^{t} \sqrt{ℓ (s)} d s < \frac{4}{λ} \sqrt{C log (\frac{1}{} δ)} .

Hence, if $n > 16 C^{2} γ_{2}^{2} (log δ)^{2} / λ^{2}$ is chosen such that

C \frac{log (1 / δ) γ_{2}}{\sqrt{n}} + \frac{4 C \sqrt{C}}{λ} \frac{\sqrt{log (\frac{1}{δ})}}{\sqrt{n}} + \frac{C^{7 / 4} \sqrt{∥ f ∥_{\infty}}}{n} {(\frac{4}{λ})}^{3 / 2} {(log (\frac{1}{δ}))}^{3 / 4} < \frac{λ}{2},

also holds, then we must have $T_{b} = \infty$ . In other words, for some $C_{1} > 0$ ,

n \geq C_{1} (γ_{2}^{2} + \sqrt{∥ f ∥_{\infty}}) (log δ)^{2}

will imply $T_{b} = \infty$ .

Otherwise, suppose $T_{b} < \infty$ . Then,

still holds. Then, by continuity, there exists $ϵ > 0$ such that on $t \in (T_{b}, T_{b} + ϵ)$ this still holds. This then contradicts with the definition of $T_{b}$ .

Now that $T_{b} = \infty$ , $ℓ (t) \leq ℓ (0) exp (- λ t / 2)$ holds for all $t$ . Lastly, we note $γ_{2} \leq C \sqrt{d}$ , the claim then follows. ∎

We now move to the corollary regarding approximation.

Proof of Corollary 1.

We fix $ϵ > 0$ . Clearly, one can use a function that is Lipschitz to approximate $f$ with small error under $L^{2}$ norm. Hence, we assume that $f$ is Lipschitz.

We first of all have the following estimates:

\begin{matrix} \int_{D} | f (x) - f^{(n)} (x) |^{2} d x \leq ∣ ∣ ∣ \int_{D} | f (x) - f^{(n)} (x) |^{2} d x - \int_{D} | f (x) - f^{(n)} (x) |^{2} d ν_{M} ∣ ∣ ∣ + \int_{D} | f (x) - f^{(n)} (x) |^{2} d ν_{M}, \end{matrix}

(3.14)

where

ν_{M} = \frac{1}{M} M \sum p = 1 δ (x - x_{p})

is the empirical measure for data ${x_{p}}_{p = 1}^{M}$ .

By the standard result of convergence empirical measures [11], there exists such $ν_{M}$ with $M ≫ 1$ such that,

W_{1} (ν_{M}, d x) < ϵ,

where $W_{1}$ is the Wasserstein distance.

Then, we set $δ = \frac{1}{2}$ . Then, we take the event such that the claims in Theorem 1 hold for $ν_{M}$ , and such that $\sum_{i = 1}^{n} w_{i}^{2} (0) \leq C$ . Hence, we are able to find the parameters $(a_{i} (t), w_{i} (t))$ for $t$ large enough such that

\int_{D} | f (x) - f^{(n)} (x, t) |^{2} d ν_{M} \leq ϵ .

The important consequence of Theorem 1 is that $n$ is independent of $ν_{M}$ (and thus $M$ ).

Now, we fix such $t > 0$ and estimate the Lipschitz constant of $| f (x) - f^{(n)} |^{2}$ . The gradient of this function is

- 2 (f (x) - f^{(n)} (x, t)) (\nabla f + \frac{1}{\sqrt{n}} n \sum i = 1 a_{i} (t) σ^{'} (w_{i} (t) \cdot x) w_{i} (t)) .

Hence, we need to estimate $| f^{(n)} (x) |$ and $| a_{i} (t) w_{i} (t) |$ .

By Lemma 3.1,

n \sum i = 1 a_{i}^{2} (t) \leq n \sum i = 1 a_{i}^{2} (0) + C = n + C .

Similarly,

| w_{i} (t) | \leq | w_{i} (0) | + \frac{C}{\sqrt{n}} \int_{0}^{t} | a_{i} (s) | \sqrt{ℓ (s)} d s .

Hence,

n \sum i = 1 | w_{i} | \leq \sqrt{n} \sqrt{n \sum i = 1 | w_{i}^{2} (0) |} + C \int_{0}^{t} \sqrt{n \sum i = 1 a_{i}^{2} (s)} \sqrt{ℓ (s)} d s \leq C \sqrt{n} .

Hence, $| a_{i} (t) | \leq C \sqrt{n}$ and $| w_{i} (t) | \leq C \sqrt{n}$ . In other words, the Lipschitz constant of $| f (x) - f^{(n)} |^{2}$ is controlled by a polynomial of $n$ and the Lipschitz constant of $f$ (but independent of $M$ ), since $n$ is independent of $M$ .

The whole error is thus given by $C ϵ + L i p (| f (x) - f^{(n)} |^{2}) ϵ$ . As $ϵ$ is arbitrary, the claim follows. ∎

4 Conclusion and discussion

Current works show the convergence of loss function in the overparametrized regime for neural networks. By considering the loss function directly, we obtain the decay of the loss function in a rate independent of the number of training data, where the approximation is given specifically by (1.3). The key observation was made by S. Du et al that the weights are close to the initialization. In this sense, the loss function has global minimizations “everywhere” in the parameter space. As we have seen, the proof works for large number of data $M \to \infty$ , and as Corollary 1

indicates, the parameters that can achieve the good approximation seems to be available ”everywhere” in the parameter space, which we strongly believe leads to the good performance of stochastic gradient descent. If we instead use the approximation of the form (

1.2) inspired by mean field limit, then one must wait the weights to converge to some specific regions to gain global minimization. This probably means the approximation (1.3) has larger capacity and may be used to explain why SGD behaves well.

The loss function used here is quadratic functions. However, we argue that quadratic functions are general enough since near the minima, all $C^{2}$ functions look like quadratic functions.

There are many instant subsequent works, for example, one may explore whether the ideas and techniques here can be applied to improve the the generalization error estimates in literature, and to get better explanation of why stochastic gradient descent works well. These should be challenging but exciting problems.

Acknowledgement

The work of L. Li was partially sponsored by NSFC 11901389 and 11971314, and Shanghai Sailing Program 19YF1421300.

Appendix A Proof of Proposition 1

The proof divides into several steps.

Step 1. Basic concentration inequalities.

Define

φ (v) := \sqrt{v + 1}

(A.1)

and define

W_{x} := φ (| Z_{x} |) .

(A.2)

Hence, using $| \sqrt{a} - \sqrt{b} | \leq \sqrt{| a - b |}$ , one has

| W_{x} - W_{y} | \leq \sqrt{| | Z_{x} | - | Z_{y} | |} \leq \sqrt{| Z_{x} - Z_{y} |} .

Note that

Z_{x} - Z_{y} = \frac{1}{n} n \sum i = 1 (Y_{x}^{i} - Y_{y}^{i}),

where ${Y_{x}^{i} - Y_{y}^{i}}$ ’s are still independent with mean zero. By Bernstein inequality [13], one has for any $x, y$

\begin{matrix} P (| Z_{x} - Z_{y} | \geq t | x - y |) & \leq 2 exp (- \frac{n t^{2} | x - y |^{2} / σ^{2}}{2 (1 + L t | x - y | / σ)}) \leq 2 exp (- C_{1} n min (t^{2}, t)), \end{matrix}

(A.3)

where $σ^{2} := E (Y_{x}^{i})^{2} \leq C | x - y |^{2}$ .

Consequently,

P (| W_{x} - W_{y} | \geq t \sqrt{| x - y |}) \leq P (| Z_{x} - Z_{y} | \geq t^{2} | x - y |) \leq 2 exp (- C_{1} n min (t^{4}, t^{2}))

Hence, one defines the metric (note that the maximum of two metrics is still a metric)

g (x, y) = max (| x - y |, | x - y |^{1 / 2}),

(A.4)

then one has

P (| Z_{x} - Z_{y} | \geq t g (x, y)) \leq 2 exp (- C_{1} n min (t^{2}, t)),

(A.5)

while for $t \geq 1$ ,

P (| W_{x} - W_{y} | \geq t g (x, y)) \leq 2 exp (- C_{1} n t^{2})

(A.6)

Step 2 Generic chaining

Now consider the domain $D$ equipped with the metric $g$ in (A.4). Define a number

γ_{α} := inf sup x \in D \infty \sum s = 0 2^{s / α} g (x, T_{s}),

(A.7)

where $T_{s} \subset D$ is a set containing $2^{2^{s}}$ points. The $inf$ is taken over the collections of such sets ${T_{s}}_{s = 0}^{\infty}$ . From $s$ to $s + 1$ , the number of points in $T_{s}$ is squared. Hence, the distance $g (x, T_{s})$ decays double exponentially if the sequence ${T_{s}}$ is chosen suitably. Hence, $γ_{α}$ is well-defined. For a given $T_{s}$ , for every $x \in D$ , there must be $y \in T_{s}$ , such that

g (x, T_{s}) = g (x, y),

and we define

π_{s} (x) := y .

(A.8)

Let $g_{e} (x, T_{s}) := {inf}_{y \in T_{s}} | x - y |$ be the usual Euclidean distance, and we find that on the suitable sequence of subsets, $\sqrt{g_{e} (x, T_{s})} \sim g_{e} (x, T_{s - 1})$ . Hence, the number $γ_{α}$ is comparable to

{~ γ}_{α} := inf sup x \in D \infty \sum s = 0 2^{s / α} g_{e} (x, T_{s}) .

Below, we focus only on $α = 2$ . The number of ${~ γ}_{2}$ for the unit ball is like $\sqrt{d}$ as shown in [10, sec. 2.2], where $d$ is the dimension.

By the definition of $γ_{2}$ , we are able to choose the collection of ${T_{s}}$ such that

\infty \sum i = 1 2^{s / 2} g (π_{s} (x), π_{s + 1} (x)) \leq 2 γ_{2},

which holds for all $x \in D$ .

Step 3 Controlling $W_{x}$ .

Let $s_{0}$ be such that $n < 2^{s_{0}} \leq 2 n$ . Consider any $s \geq s_{0}$ . By step 1, for any fixed $y \in T_{s}, x \in T_{s + 1}$ , one has

\begin{matrix} P ⎛ ⎝ | W_{x} - W_{y} | \geq \frac{2}{C_{1}} \sqrt{\frac{log (| T_{s} | | T_{s + 1} |)}{n}} g (x, y) ⎞ ⎠ \leq 2 exp (- 2 log (| T_{s} | | T_{s + 1} |)) = \frac{2}{| T_{s} |^{2} | T_{s + 1} |^{2}} \leq \frac{1}{| T_{s} |^{2} | T_{s + 1}} \end{matrix}

On the convergence of gradient descent for two layer neural networks

Authors

Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks

Refined Generalization Analysis of Gradient Descent for Over-parameterized Two-layer Neural Networks with Smooth Activations on Classification Problems

On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamics

Plateau Phenomenon in Gradient Descent Training of ReLU networks: Explanation, Quantification and Avoidance

When does gradient descent with logistic loss find interpolating two-layer networks?

Beyond Lazy Training for Over-parameterized Tensor Decomposition

Stochastic Markov Gradient Descent and Training Low-Bit Neural Networks

1 Introduction

2 Mathematical setup and the main result

Assumption 1.

Assumption 2.

Assumption 3.

Theorem 1.

Corollary 1.

3 The proof of the main result

Lemma 1.

Lemma 2.

Proof.

Proposition 1.

Lemma 3.

Proof.

Lemma 4.

Lemma 5.

Proof.

Proof of Theorem 1.

Proof of Corollary 1.

4 Conclusion and discussion

Acknowledgement

Appendix A Proof of Proposition 1

On the convergence of gradient descent for two layer neural networks

Authors

Related Research

Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks

Refined Generalization Analysis of Gradient Descent for Over-parameterized Two-layer Neural Networks with Smooth Activations on Classification Problems

On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamics

Plateau Phenomenon in Gradient Descent Training of ReLU networks: Explanation, Quantification and Avoidance

When does gradient descent with logistic loss find interpolating two-layer networks?

Beyond Lazy Training for Over-parameterized Tensor Decomposition

Stochastic Markov Gradient Descent and Training Low-Bit Neural Networks

This week in AI

1 Introduction

2 Mathematical setup and the main result

Assumption 1.

Assumption 2.

Assumption 3.

Theorem 1.

Corollary 1.

3 The proof of the main result

Lemma 1.

Lemma 2.

Proof.

Proposition 1.

Lemma 3.

Proof.

Lemma 4.

Lemma 5.

Proof.

Proof of Theorem 1.

Proof of Corollary 1.

4 Conclusion and discussion

Acknowledgement

Appendix A Proof of Proposition 1