Gradient Descent Converges to Ridgelet Spectrum

1 Introduction

Characterizing the local minima of the gradient descent (GD) learning is important for the theoretical study of neural networks. Due to the non-convexity of the learning problem, it is a hard and challenging problem. The over-parametrization, an assumption that the parameter number is sufficiently larger than the sample size, is considered to be an important factor to prove the better performance of deep learning (Arora et al., 2019b; Allen-Zhu et al., 2019; Nitanda and Suzuki, 2017; Rotskoff and Vanden-Eijnden, 2018; Mei et al., 2018; Sirignano and Spiliopoulos, 2020a; Chizat and Bach, 2018; Jacot et al., 2018; Lee et al., 2019; Frankle and Carbin, 2019). By regarding the weights of parameters in a neural network as a signed distribution, we analyze the over-parametrized regime by means of the integral representation (Barron, 1993; Murata, 1996; Sonoda and Murata, 2017): $S [γ] := \int γ (a, b) σ (a \cdot x - b) d a d b$ . Here, $γ$ represents the weights, or the parameter distribution, and $σ (a \cdot x - b)$

represents a hidden unit with an activation function

σ

, input

x

and hidden parameters

(a, b)

. This is a weighted integral of infinite hidden units, but we remark that by formally letting a singular measure

\sum_{i = 1}^{p} c_{i} δ_{(a_{i}, b_{i})}

γ

, we can also represent a weighted sum of finite hidden units as

\sum_{i = 1}^{p} c_{i} σ (a_{i} \cdot x - b_{i})

Since all the hidden parameters $(a, b)$ are integrated out, we do not need to update hidden parameters during the training, and we only need to update the parameter distribution $γ$ . This is a strong advantage because the learning problem regains the convexity in the function space. This convexification trick has been known and employed in the integral representation theory (Barron, 1993) a.k.a. ridgelet analysis (Murata, 1996; Candès, 1999), convex neural networks (Bengio et al., 2006) and random Fourier features (Rahimi and Recht, 2008). Recently, the mean-field regime (Mei et al., 2018; Rotskoff and Vanden-Eijnden, 2018; Sirignano and Spiliopoulos, 2020a), a.k.a. the Wasserstein gradient flow theory (Chizat and Bach, 2018; Nitanda and Suzuki, 2017) also adopted this formulation, and the integral representation has been recognized as a crucial tool to prove the global convergence of deep learning.

Thus far, we know less about the minimizers themselves, due to the non-trivial null space of the integral operator $S$ . In fact, there are infinitely many different parameter distributions, say $γ_{1}$ and $γ_{2}$ , that indicate the same function: $S [γ_{1}] = S [γ_{2}]$ (see Appendix C.3 for more details). In this study, we consider a regularized square loss minimization problem and provide a unique explicit representation of the global minimizer in terms of the ridgelet transform on the torus.

The ridgelet transform, which is a wavelet-like integral transform, is originally developed in (Murata, 1996; Candès, 1998; Sonoda and Murata, 2017)

, and has a remarkable application to analysis of neural networks. Whereas in the original studies of the ridgelet transform, they employed the Fourier analysis on the Euclidean space, we utilize the Fourier analysis on the torus, and develop a simple but flexible framework to study the neural networks with modern activation functions such as the rectified linear unit (ReLU). Although the Fourier analysis on the torus imposes the periodicity on the activation function, we theoretically show a periodic activation function still provides a sufficient and effective power to analyze the over-parametrized neural networks.

To be precise, our main theorem is described as follows:

Main Theorem.

Let $F := L^{2} (R^{m}, P)$ be the space of data generating functions ( $P$ : finite Borel measure with density $q$ ) and $G := L^{2} ([- A, A]^{m} \times [- T / 2, T / 2]; d a d b)$ the space of parameter distributions. Here $A$ and $T$ be a bounds of the hidden parameters. We assume the activation function $σ$ is periodic with period $T$ . For any $f \in F$ and $β > 0$ , the unique solution of the following minimization problem

γ_{A, β}^{*} [f] := a r g m i n γ \in G {∥ ∥ ∥ f - \int γ (a, b) σ (a \cdot x - b) d a d b ∥ ∥ ∥}_{F}^{2} + β ∥ γ ∥_{G}^{2} .

(1)

is uniquely represented by the ridgelet transform:

	$γ_{A, β}^{*} [f] =$	$\int_{R^{m}} \frac{f (x) q (x)}{q (x) + β} σ (a \cdot x - b) d x$
		$+ (residual terms depending only on P, A and β) .$		(2)

Here, the residual terms tend to $0$ as $A \to \infty$ .

Numerical simulation confirms our main results, namely, the scatter plot of parameter distributions learned by GD shows a similar pattern to the ridgelet spectrum. As a consequence, we can also gain a better understanding of the theoretical background of lazy learning, a recent trend of inductive bias theory stating that the learned parameters are very close to the initial parameters, such as the neural tangent kernel (Jacot et al., 2018; Lee et al., 2019; Arora et al., 2019a) and the strong lottery ticket hypothesis (Frankle and Carbin, 2019).

The structure of this paper is as follows: In Section 2, we develop the theory of the ridgelet transform on the torus. It is a theoretical basis to provide an explicit representation of the over-parametrized neural network at the global minimum. Then we introduce a positive definite kernel to prove universality of neural networks with a bounded range of hidden parameters, and give a precise definition of integral representations. In Section 3, we give our main results. In Section 4, we conduct numerical simulation to see that the ridgelet spectrum is identical with the parameter distribution of over-parameterized neural networks trained by gradient descent, as suggested in our main works. In Section 5, we discuss the relation to previous studies.

Notation.

For a measurable space $X$ with a positive measure $μ$ , we denote by $L^{2} (X, μ)$ the square integrable fuoncions on $X$ with respect to $μ$ .

2 Ridgelet Transforms on the Torus

In this section, we establish the ridgelet transform on the torus, which is a theoretical basis of this study. We fix $T > 0$ , and denote by $T$ the torus $R / T Z$ , and often regard $T$ as the interval $[- T / 2, T / 2)$ . We fix a bounded measurable function $σ : T \to R$ , or equivalently, a bounded measurable periodic function $σ$ on $R$ with period $T$ ( $σ (x + T) = σ (x)$ ). For an integer $n$ , we write $ˆ σ (n) := (1 / T) \int_{- T / 2}^{T / 2} σ (x) e^{2 π i n x / T}$ . Originally, the ridgelet transform has been defined on the Euclidean space (Murata, 1996; Candès, 1999). However, the original definition excludes non-integrable activation functions such as Tanh and ReLU. Sonoda and Murata (Sonoda and Murata, 2017) have extended the ridgelet transform to accept such non-integrable activation functions, by introducing an auxiliary dual activation function. However, their theory sacrifices the Plancherel formula, which we need in this study. Therefore, in order to cover the non-integrable activation functions, we come to suppose periodic activation functions.

2.1 Ridgelet transform

We introduce the ridgelet transform and its reconstruction formula.

Definition 2.1.

We define the ridgelet transform $R : L^{2} (R^{m}, d x) \to L^{2} (R^{m} \times T, d a d b)$ by

R [f] (a, b) := \int_{R^{m}} f (x) σ (a \cdot x - b) d x .

(3)

To be precise, we define $R [f]$ for all the $f \in L^{2} (R^{m})$ via bounded extension, an essentially the same arguments in the definition of the $L^{2}$

-Fourier transform. Namely, We first define

R [f]

for

f \in L^{1} (R^{m})

, which is absolutely convergent because

σ \in L^{\infty} (T)

. Then, we extend

R [f]

for

f \in L^{2} (R^{m})

as a common limit of

R [f_{i}]

, where

f_{i}

is any sequence in

L^{1} (R^{m}, d x) \cap L^{2} (R^{m}, d x)

that converges to

f

L^{2}

. Let us introduce the admissible condition on

σ

Assumption 2.2 (admissible condition).

The function $σ \in L^{\infty} (T)$ satisfies the following two conditions: (1) $ˆ σ (0) = 0$ , and (2) $\sum_{n \neq 0} | ˆ σ (n) |^{2} / | n |^{m} = T^{- m - 1}$ .

We need the admissibility condition in the proof of the reconstruction formula blow. It is not at all strong. In fact, the infinite sum of the second condition always converge because $σ$ is square integrable, thus, we may replace $σ$ with a function satisfying these condition via only multiplying and subtracting constants. In particular, restrictions of Tanh and ReLU to $T$ can satisfy this assumption with slight modifications on the constants. Under the admissible condition, the ridgelet transform meets the reconstruction formula and the Plancherel formula as follows:

Theorem 2.3.

Impose Assumption 2.2 on $σ$ . Then for $f, g \in L^{2} (R^{m}, d x)$ , we have

	$\int_{R^{m} \times T} R [f] (a, b) σ (a \cdot x - b) d a d b$	$= f (x),$		(4)
	$⟨ R [f], R [g] ⟩_{L^{2} (R^{m} \times T, d a d b)}$	$= ⟨ f, g ⟩_{L^{2} (R^{m}, d x)},$		(5)

By the Plancherel formula (5), the adjoint operator of $R$ is calculated as $R^{*} [γ] (x) = \int γ (a, b) σ (a \cdot x - b) d a d b$ for $γ \in I m (R)$ . This might be regarded as an integral representation of the neural network. However, this integral transform $R^{*}$ is defined only on the image of $R$ , which is hard to specify (due to the non-triviality of the null space $ker S$ ). Thus, we will introduce the modified version of it in Section 2.3. By discretizing the integral in (4), we have a well-known universality in $L^{2} (R^{m}, P)$ ( $P$ is a finite Borel measure) of 2-layer neural networks with the activation $σ$ as a corollary of Theorem 2.3:

Corollary 2.4.

For any finite Borel measure $P$ on $R^{m}$ , the linear space generated by ${σ_{a, b} : (a, b) \in R^{m} \times T}$ is dense in $L^{2} (R^{m}, P)$ .

2.2 Reproducing kernel Hilbert space with inner product of features

In this section, we introduce an RKHS, which is an effective framework to analyze behaviors of the parameters and expressive power of neural networks, and we prove a stronger universality result (Corollary 2.7) for 2-layer neural networks with parameters restriction for later use.

We fix a positive number $A > 0$ and let $I_{A} := [- A, A]$ . We define a positive definite kernel on $R^{m}$ by

k_{A} (x, y) := \int_{I_{A}^{m} \times T} σ (a \cdot x - b) σ (a \cdot y - b) d a d b .

(6)

We denote by $H$ the RKHS associated with the kernel $k_{A}$ . We call $H$ the RKHS with inner product of features. We remark that $k_{A}$ is a continuous and bounded kernel. Next we discuss the characteristic property and $c_{0}$ -universality ((Sriperumbudur et al., 2010, p.2392)) of $H$ , namely, density properties in function spaces. To deal with this problem, let us introduce the following mild assumption on $σ$ :

Assumption 2.5.

The bounded measurable function $σ$ on $T$ satisfies $# {n \in Z : ˆ σ (n) \neq 0} = \infty$ .

In other words, the $σ$ cannot be a finite sum of trigonometric polynomials, and thus any discontinuous square integrable function on $T$ satisfies this assumption. For example, $R e L U |_{[- T / 2, T / 2]}$ and $tanh |_{[- T / 2, T / 2]}$ satisfy this. Under this assumption we have the following theorem:

Theorem 2.6.

Under Assumption 2.5, the $k_{A}$ is characteristic. If we additionally impose Assumption 2.2 on $σ$ , $k_{A}$ is $c_{0}$ -universal.

By means of Theorem 2.6, we prove a stronger form of universality as follows:

Corollary 2.7.

For any finite Borel measure $P$ on $R^{m}$ and $A > 0$ , the linear space generated by ${σ_{a, b} : (a, b) \in I_{A}^{m} \times T}$ is dense in $L^{2} (R^{m}, P)$ . Here we define $σ_{a, b} (x) := σ (a \cdot x - b)$ .

Proof.

We here denote by $⟨, ⟩$ the inner product in $L^{2} (R^{m}, P)$ . It suffices to show that for any $f \in L^{2} (R^{m}, P)$ , $f = 0$ if $⟨ f, σ_{a, b} ⟩ = 0$ for all $(a, b) \in I_{A} \times T$ . Since $σ_{0, b}$ is a nonzero constant function for a $b \in T$ , we see that $⟨ f, 1 ⟩ = 0$ . Let $k_{A, y} (x) := k_{A} (x, y)$ . Then we have $⟨ f, k_{A, y} ⟩ = \int_{I_{A}^{m} \times T} ⟨ f, σ_{a, b} ⟩ σ_{y} (a, b) d a d b = 0$ for all $y \in R^{m}$ where $σ_{x} (a, b) := σ (a \cdot x - b)$ . Since $H$ is generated by $k_{A, x}$ ’s, thus we conclude that $f$ is contained in the orthogonal complement of $R + H$ . Since $k_{A}$ is characteristic, the space $R + H$ is dense in $L^{2} (R, P)$ (cf. (Kenji Fukumizu and Jordan, 2009, Proposition 5), (Sriperumbudur et al., 2010, Section 3.2)). Hence we have $f = 0$ . ∎

Compared with Proposition 2.4, Corollary 2.7 provides a practically important conclusion, namely, it says even if the parameters in the hidden layer are bounded, 2-layer neural networks have sufficient expressive power under Assumption 2.5.

2.3 Integral representation of neural networks

In this section, we define an integral representation of a 2-layer neural network. It is also regarded as a truncated version of the adjoint operator $R^{*}$ of the ridgelet transform $R$ . Although the theory of the ridgelet transform on $R^{m} \times T$ is very clear, it has a flaw to analyze the neural networks. In fact, because $L^{2} (R^{m}, d x)$ does not contains $σ_{a, b} (x) := σ (a \cdot x - b)$ , thus any finite neural networks, we cannot see the direct connection between finite neural networks and integral representations of neural networks. To circumvent this technical issue, we consider a $P$ -weighted version (since $σ_{a, b} \in L^{2} (R^{m}, P)$ ).

Definition 2.8.

We define an integral representation of a neural network $S_{A} : L^{2} (I_{A}^{m} \times T, d a d b) \to L^{2} (R^{m}, P)$ by

S_{A} [γ] (x) := \int_{I_{A}^{m} \times T} γ (a, b) σ (a \cdot x - b) d a d b .

(7)

The operator $S_{A}$ can be regarded as a limit of neural networks of the form $\sum c_{i} σ (a_{i} \cdot x - b_{i})$ whose hidden parameters $(a_{i}, b_{i})$ are contained in $I_{A}^{m} \times T$ . By simple computation, we see that the adjoint operator $S_{A}^{*}$ is explicitly represented as $S_{A}^{*} [f] (a, b) = \int f (x) σ (a \cdot x - b) d P (x)$ . Thus $S_{A}$ is the adjoint operator of a weighted analogue of the ridgelet transform (c.f. Definition 2.1). Then we have the following proposition describing the expressive power of $S_{A}$ :

Proposition 2.9.

Assume $σ$ is continuous at a point $b_{0}$ and $σ (b_{0}) \neq 0$ . The image of $S_{A}$ is dense in $L^{2} (R^{m}, P)$ .

Proof.

Denote by $⟨, ⟩$ the inner product of $L^{2} (R^{m}, P)$ . Since the image of $S_{A}$ is the same as the orthogonal complement of the adjoint operator $S_{A}^{*}$ , it suffices to show that $S_{A}^{*} [f] = 0$ implies $f = 0$ . In fact, if $S_{A}^{*} [f] = 0$ , then $S_{A}^{*} [f] (a, b) = ⟨ f, σ_{a, b} ⟩ = 0$ for almost every $(a, b)$ . Since $S_{A}^{*} [f]$ is continuous on ${a \neq 0}$ , we see that $⟨ f, σ_{a, b} ⟩ = 0$ for all $(a, b)$ with $a \neq 0$ . In addition, $0 = {lim}_{| a | \to 0} S_{A}^{*} [f] (a, b_{0}) = σ (b_{0}) ⟨ f, 1 ⟩$ . Thus by Corollary 2.7, we have $f = 0$ . ∎

3 Main Results

In this section, we describe the formulation of our problem and main results. We impose Assumptions 2.2 and 2.5 on the bounded measurable map $σ$ on $T$ . We fix an absolutely continuous finite Borel measure $P$ on $R^{m}$ . We assume $P$ has a bounded density function $q$ .

3.1 Square loss minimization for the integral representation

Our main goal is to provide an explicit representation of the global minimizer of the learning problem:

min γ {∥ f - S_{A} [γ] ∥}_{A}^{2} .

(8)

We regard the function $γ \in L^{2} (I_{A}^{m} \times T, d a d b)$ as a distribution $γ (a, b) d a d b$ . We will give an explicit representation to the distribution attaining the solution of $(???)$ in terms of the ridgelet transform.

For $f \in L^{2} (R^{m}, P)$ , $A > 0$ , and $β > 0$ , we consider the $L^{2}$ -regularized square loss of the integral representation $S_{A}$ :

L_{β, A} (γ; f) := {∥ f - S_{A} [γ] ∥}_{A}^{2} + β {∥ γ ∥}_{L^{2} (I_{A}^{m} \times T, d a d b)}^{2} .

(9)

We denote by $γ_{β, A}^{*} [f]$ the unique element that attains ${min}_{γ} L_{β, A} (γ; f)$ , which always exists as long as $S_{A}$ is densely defined closed operator. See Appendix D for more details. Although ${∥ γ ∥}_{L^{2} (I_{A}^{m} \times T, d a d b)}^{2}$ can tend to infinity as $β \to 0$ , by Proposition 2.9, we have the following proposition:

Proposition 3.1.

Assume $σ$ is continuous at a point $b_{0}$ and $σ (b_{0}) \neq 0$ . Then the square loss $∥ ∥ f - S_{A} [γ_{β, A}^{*} [f]] {∥ ∥}_{L^{2} (R^{m}, P)}^{2}$ converges to 0 as $β \to 0$ .

Proof.

Let $ε > 0$ be an arbitrary positive number. By Proposition 2.9, there exists an element of $L^{2} (I_{A}^{m} \times T, d a d b)$ such that ${∥ f - S_{A} [γ] ∥}_{A}^{2} < ε / 2$ . Take $β = ε / 2 {∥ γ ∥}_{L^{2} (I_{A}^{m} \times T, d a d b)}^{2}$ . Then $γ_{β, A}^{*} [f]$ satisfies $ε > L_{β, A} (γ_{β, A}^{*} [f]; f) > ∥ ∥ f - S_{A} [γ_{β, A}^{*} [f]] {∥ ∥}_{L^{2} (R^{m}, P)}^{2}$ . Thus we have $∥ ∥ f - S_{A} [γ_{β, A}^{*} [f]] {∥ ∥}_{L^{2} (R^{m}, P)}^{2} \to 0$ . ∎

3.2 An explicit representation of the global minimizer

Our first main result is the explicit representation of the minimizer of the regularized square loss minimization problem in terms of the ridgelet transform.

Theorem 3.2.

Let $P$ be an absolutely continuous finite Borel measure on $R^{m}$ with bounded density function $q$ . Let $f \in L^{2} (R^{m}, d x)$ . Then, automatically $f \in L^{2} (R^{m}, P)$ and we have

γ_{β, A}^{*} [f] = R [\frac{q f}{β + q}] + Δ_{β, A} [f],

(10)

where $Δ_{β, A} [f]$ is an element of $L^{2} (I_{A}^{m} \times T, d a d b)$ such that

lim A \to \infty {∥ ∥ Δ_{β, A} [f] ∥ ∥}_{L^{2} (I_{A}^{m} \times T, d a d b)} = 0.

(11)

By formally completing square in the Hilbert space, we can verify the unique existence of the minimizer $γ_{β, A}^{*}$ . However, the concrete property of $γ_{β, A}^{*}$ is not clear. Theorem 3.2 provides the explicit representation of the minimizer via ridgelet transform.

3.3 Relation to the 2-layer finite neural networks

In this section, we prove that the over-parametrized finite neural networks converge to the minimizer in the integral representation (9) as the parameter number $p$ tends to infinity. The over-parametrization may let us suppose that the parameter distribution $λ_{p} := p^{- 1} \sum_{i = 1}^{p} δ_{(a_{i}, b_{i})}$ of a neural network $g_{p} (x) := \sum_{i = 1}^{p} c_{i} σ (a_{i} \cdot x - b_{i})$ weakly converges

to the uniform distribution

U

I_{A}^{m} \times T

, and the problem reduces to the optimization of

{c_{i}}_{i = 1}^{p}

. Here, the weak convergence assumption is satisfied, for example, when the parameters

{(a_{i}, b_{i})}_{i = 1}^{\infty}

are i.i.d. samples drawn from

U

. However, the randomness is not necessary in the proof. Let us consider the supervised learning problem as follows: Given a sequence

{λ_{p}}_{p \in N}

of Borel measures that weakly converges to the Lebesgue measure

d a d b

I_{A}^{m} \times T

, define

S_{p} : L^{2} (I_{A}^{m} \times T, λ_{p}) \to L^{2} (R^{m}, P)

by allocating

\int γ (a, b) σ (a \cdot x - b) d λ_{p} (a, b) = \sum_{i = 1}^{p} γ (a_{i}, b_{i}) σ (a_{i} \cdot x - b_{i})

γ \in L^{2} (I_{A}^{m} \times T, λ_{p})

. For

β > 0

, we consider the following optimization problem:

min γ \in L^{2} (λ_{p}) {∥ f - S_{p} [γ] ∥_{L^{2} (R^{m}, P)}^{2} + β ∥ γ ∥_{L^{2} (I_{A}^{m} \times T, λ_{p})}^{2}}

(12)

Let $γ_{β, A}^{p}$ be the unique minimizer of (12).

Theorem 3.3.

Assume ${λ_{p}}_{p = 1}^{\infty}$ weakly converges to $U$ as $p \to \infty$ . Then the minmimizer $γ_{β, A}^{p}$ of (12) converges to the minimizer $γ_{β, A}^{*}$ of (9) in the sense $∥ γ_{β, A}^{p} - γ_{β, A}^{*} ∥_{L^{2} (I_{A}^{m} \times T, λ_{p})} \to 0$ as $p \to \infty$ .

Although we cannot catch the shape of the distribution of the optimal solution $γ_{β, A}^{p}$ when the parameter number $p$ is small, the over-parametrized neural networks converge to a common parameter distribution. Combining Theorem 3.3 with Theorem 3.2, we obtain an explicit representation of the global minimizer via the ridgelet transform. Theorem 3.3 implies the weak convergence of parameter distributions, which is a stronger convergence of over-parametrized neural networks to the global minimum than previous results:

Corollary 3.4.

The distribution $γ_{β, A}^{p} d λ_{p}$ weakly converges to an absolutely continuous distribution $γ_{β, A}^{*} d a d b$ , namely, for any bounded continuous function $f$ on $I_{A}^{m} \times T$ , we have $\int f γ_{β, A}^{p} d λ_{p} \to \int f γ_{β, A}^{*} d a d b$ as $p \to \infty$ .

In Section 4 below, we consider a learnng problem a 2-layer neural network via gradient descent. We see the parameters of the over-parametrized neural networks accumulate the ridgelet spectrums.

4 Numerical Simulation

In order to verify the main results, we conducted numerical simulation with artificial datasets. Here, we only display the results of Experiment 1. The readers are also encouraged to refer supplementary materials for further experimental results.

4.1 Scatter plots of GD trained parameters.

Given a dataset $D_{n} = {(x_{i}, y_{i})}_{i = 1}^{n}$ , we repeatedly trained $s = 1, 000$ neural networks $g (x; θ^{(t)}) = \sum_{j = 1}^{p} c_{j}^{(t)} σ (a_{j}^{(t)} x - b_{j}^{(t)}), (t \in [s])$ with activation function $σ =$ Gaussian, Tanh and ReLU. The training is conducted by minimizing the square loss: $L (θ) = \frac{1}{n} \sum_{i = 1}^{n} | y_{i} - g (x_{i}; θ) |^{2}$

using stochastic gradient descent with weight decay. Note that weight decay has an equivalent effect to

L^{2}

regularization, which we assumed in the main theory. After the training, we obtained

s p

sets of parameters

{(a_{j}^{(t)}, b_{j}^{(t)}, c_{j}^{(t)})}_{t \in [s], j \in [p]}

, and plotted them in the

(a, b, c)

-space. (

c

is visualized in color.) See supplementary materials for more details on the settings.

4.2 Ridgelet spectrum

Given a dataset $D_{n} = {(x_{i}, y_{i})}_{i = 1}^{n}$ , we approximately compute the ridgelet spectrum $R [f] (a, b)$ of $f$ at every sample points $(a, b)$ by numerical integration:

R [f] (a, b) \approx \frac{1}{n} n \sum i = 1 y_{i} σ (a x_{i} - b) Δ x,

(13)

where $Δ x$ is a normalizing constant, which is a constant because we assume that $x_{i}$ be uniformly distributed. We remark that more sophisticated methods for the numerical computation of the ridgelet transform has been developed. See (Do and Vetterli, 2003) and (Sonoda and Murata, 2014) for example.

4.3 Results

In Figure 1, we have compared the scatter plot of gradient descent (GD) trained parameters and the ridgelet spectra. All six figures are obtained from the common data generating function $f (x) = sin 2 π x$ on $[- 1, 1]$ . Despite the fact that the scatter plot and ridgelet spectrum are obtained from different procedures: numerical optimization and numerical integration, both figures share characteristics in common. For example, red and blue parameters in the scatter plots (a-c) concentrate in the area where the ridgelet spectra (d-f) indicate the same colors. Due to the periodic assumption, the ridgelet spectrum spreads infinitely in $b$ with period $T = 1$ . On the other hand, due to the weight decay assumption and initialized locations of parameters, the GD trained parameters gathers around the origin. Here, we used the uniform distribution $U (- 1, 1)$ for the initialization. We can understand that these differences between the scatter plot and the spectrum as the residual term $Δ_{A, β}$ in the main theorem. Another remarkable fact is that the GD trained parameters essentially did not change their positions in $(a, b)$ from the initialized value. This is possible because the support of initial parameters overlap the ridgelet spectrum from the beginning. We can understand this phenomenon as the so-called lazy regime.

5 Related Works

In the past, many authors have investigated the local minima of deep learning. However, these results have often posed strong assumptions such as that (A1) the activation function is limited to linear or ReLUs (Kawaguchi, 2016; Soudry and Carmon, 2016; Nguyen and Hein, 2017; Hardt and Ma, 2017; Lu and Kawaguchi, 2017; Yun et al., 2018); (A2) the parameters are random (Choromanska et al., 2015; Poole et al., 2016; Pennington et al., 2018; Jacot et al., 2018; Lee et al., 2019; Frankle and Carbin, 2019)

; (A3) the input is subject to normal distribution

(Brutzkus and Globerson, 2017); or (A4) the target functions are low-degree polynomials or another sparse neural network (Yehudai and Shamir, 2019; Ghorbani et al., 2019). Due to these simplifying assumptions, we know very little about the minimizers themselves. In this study, from the perspective of functional analysis, we present a stronger characterization of the distribution of parameters in the over-parametrized setting. As a result, our theory (A1’) accepts a wide range of activation functions, (A2’) need not assume the randomness of parameter distributions, (A3’) need not specify the data distribution, and (A4’) preserves the universal approximation property of neural networks such as the density in

L^{2}

The mean-field regime theory (Rotskoff and Vanden-Eijnden, 2018; Mei et al., 2018; Sirignano and Spiliopoulos, 2020a, b) a.k.a. the gradient flow theory (Nitanda and Suzuki, 2017; Chizat and Bach, 2018; Arbel et al., 2019) has also employed the integral representation and parameter distribution to prove the global convergence. These lines of studies claim that for the stochastic gradient descent learning of 2-layer networks, the “time evolution” of a discrete parameter distribution, say $γ_{p} (t)$ , with parameter number $p$ and continuous training time $t$ , asymptotically converges to the time evolution of the continuous parameter distribution as $p \to 0$

. Here, the time evolution is described by a gradient flow (the partial differential equation, the Wasserstein gradient flow, or the McKean-Vlasov equation)

\frac{d}{d t} γ_{\infty} (t) = - \frac{1}{2} \nabla_{γ} ∥ f - S [γ_{\infty} (t)] ∥^{2}

with initial condition

γ_{\infty} (0) = γ_{i n i t}

. However, we should point out that the convergence in this argument is weaker than our result. As we explained in Appendix C.3, the equation

f = S [γ]

has an infinitely different solutions, say

γ_{1}

and

γ_{2}

that satisfy

S [γ_{1}] = S [γ_{2}]

but

γ_{1} \neq γ_{2}

. Hence, even though

∥ S [γ_{p}] - S [γ_{\infty}] ∥ \to 0

, we cannot expect

∥ γ_{p} - γ_{\infty} ∥ \to 0

in general, which leaves the parameter distribution indeterminate. In contrast, by explicitly posing a regularization term, we have specified the parameter distribution of the global minimizer and shown the norm convergence in the space of parameter distributions:

∥ γ_{p} - R [f] ∥ \to 0

In order to avoid potential confusions, we provide supplementary explanations on the trick behind the mean-field theory. In the mean-field theory, the gradient flow $d γ (t) / d t = - \nabla ∥ f - S [γ (t)] ∥^{2}$ is often explained as the system of interacting particles by identifying the parameters ${(a_{i}, b_{i})}_{i = 1}^{p}$ as the coordinate system of $p$ physical particles. The particles obeys a non-linear equation of motion with interacting potential $I (a, b) := \int U (a, b; a^{'}, b^{'}) d γ (a^{'}, b^{'})$ , where $U (a, b; a^{'}, b^{'}) := \int σ (a \cdot x - b) σ (a^{'} \cdot x - b^{'}) d P (x)$

, which is derived simply by expanding the squared loss function. Based on this physical analogy, we may accept this potential as natural. However, this is the trick because by simply changing the order, we can verify that the null space

ker S

is eliminated by implicitly applying

S

in the potential:

I (a, b) = \iint σ (a \cdot x - b) σ (a^{'} \cdot x - b^{'}) d P (x) d γ (a^{'}, b^{'}) = \int σ (a \cdot x - b) S [γ] (x) d P (x) .

(14)

This clearly indicates that the interactive potential is degenerate in $γ$ , (try $γ = γ_{1}$ and $γ_{2}$ for example,) and this it the trick why the mean-field theory cannot show the stronger convergence.

The lazy learning, such as the neural tangent kernel (Jacot et al., 2018; Lee et al., 2019; Arora et al., 2019a) and the strong lottery ticket hypothesis (Frankle and Carbin, 2019), has employed a different formulation of the over-parametrization to investigate the inductive bias of deep learning. These lines of studies draw much attention by radially claiming that the minimizers are very close to the initialized state. In this study, we revealed that the shape of the parameter distribution is identified with the ridgelet spectrum. From this perspective, the lazy learning is reasonable when the initial parameter distribution covers the ridgelet spectrum in its support, because the initial parameters need not to be actively updated. Furthermore, this assumption can be reasonable because the initial parameter distribution is typically a normal (or sometimes a uniform) distribution centered at the origin $(a, b) = (0, 0)$ ; and if the data generating function $f$ is a low frequency function, then the ridgelet spectrum $R [f]$ concentrates at the origin.

6 Conclusion

In this study, we have derived the unique explicit representation—the ridgelet spectrum with residual—of the over-parametrized neural network at the global minimum. To the present, many studies have proven the global convergence of deep learning. However, we know very little about the minimizer itself because (1) the settings are typically very simplified and (2) the integral representation operator $S$ has a non-trivial null space. To circumvent these problems, we develop the ridgelet transform on the torus and conduct analysis on the regularized square loss minimization. In the numerical simulation, the scatter plots of learned parameters have shown a very similar pattern to the ridgelet spectra, which supports our theoretical result.

Broader Impact

We believe this section is not applicable to this paper because of the theoretical nature of this study.

Acknowledgments

The authors are grateful to Taiji Suzuki, Atsushi Nitanda, Kei Hagihara, Yoshihiro Sawano, Takuo Matsubara, and Noboru Murata for productive comments. This work was supported by JSPS KAKENHI 18K18113.

References

Allen-Zhu et al. (2019) Z. Allen-Zhu, Y. Li, and Y. Liang. Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers. In Advances in Neural Information Processing Systems 32, pages 6155–6166, 2019.
Arbel et al. (2019) M. Arbel, A. Korba, A. SALIM, and A. Gretton. Maximum Mean Discrepancy Gradient Flow. In Advances in Neural Information Processing Systems 32, pages 6481–6491, 2019.
Arora et al. (2019a) S. Arora, S. S. Du, W. Hu, Z. Li, R. Salakhutdinov, and R. Wang. On Exact Computation with an Infinitely Wide Neural Net. In Advances in Neural Information Processing Systems 32, pages 8139–8148, 2019a.
Arora et al. (2019b) S. Arora, S. S. Du, W. Hu, Z. Li, and R. Wang. Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks. In Proceedings of the 36th International Conference on Machine Learning
, volume 97, pages 322–332, 2019b.
Barron (1993) A. R. Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information Theory, 39(3):930–945, 1993.
Bengio et al. (2006) Y. Bengio, O. Delalleau, and N. L. Roux. The Curse of Highly Variable Functions for Local Kernel Machines. In Advances in Neural Information Processing Systems 18, pages 107–114, Vancouver, BC, 2006. MIT Press.
Brutzkus and Globerson (2017) A. Brutzkus and A. Globerson. Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputs. In Proceedings of The 34th International Conference on Machine Learning, volume 70, pages 605–614, 2017.
Candès (1998) E. J. Candès. Ridgelets: theory and applications. PhD thesis, Standford University, 1998.
Candès (1999) E. J. Candès. Harmonic analysis of neural networks. Applied and Computational Harmonic Analysis, 6(2):197–218, 1999.
Chizat and Bach (2018) L. Chizat and F. Bach. On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport. In Advances in Neural Information Processing Systems 32, pages 3036–3046, 2018.
Choromanska et al. (2015) A. Choromanska, Y. LeCun, and G. Ben Arous. Open Problem: The landscape of the loss surfaces of multilayer networks. In The 28th Annual Conference of Learning Theory, volume 40, pages 1–5, 2015.
Do and Vetterli (2003) M. N. Do and M. Vetterli. The finite ridgelet transform for image representation. Image Processing, IEEE Transactions on, 12(1):16–28, 2003.
Frankle and Carbin (2019) J. Frankle and M. Carbin. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. In International Conference on Learning Representations, pages 1–42, 2019.
Ghorbani et al. (2019) B. Ghorbani, S. Mei, T. Misiakiewicz, and A. Montanari. Limitations of Lazy Training of Two-layers Neural Network. In Advances in Neural Information Processing Systems 32, pages 9111–9121, 2019.
Hardt and Ma (2017) M. Hardt and T. Ma. Identity Matters in Deep Learning. In International Conference on Learning Representations 2017, pages 1–14, 2017.
Jacot et al. (2018) A. Jacot, F. Gabriel, and C. Hongler. Neural Tangent Kernel: Convergence and Generalization in Neural Networks. In Advances in Neural Information Processing Systems 31, pages 8571–8580, 2018.
Kawaguchi (2016) K. Kawaguchi. Deep Learning without Poor Local Minima. In Advances in Neural Information Processing Systems 29, pages 586–594, 2016.
Kenji Fukumizu and Jordan (2009) F. R. B. Kenji Fukumizu and M. I. Jordan. Kernel dimension reduction in regression. Annals of Statistics, 37(4):1871–1905, 2009.
Lee et al. (2019) J. Lee, L. Xiao, S. S. Schoenholz, Y. Bahri, J. Sohl-Dickstein, and J. Pennington. Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent. In Advances in Neural Information Processing Systems 32, pages 8572–8583, 2019.
Lu and Kawaguchi (2017) H. Lu and K. Kawaguchi. Depth Creates No Bad Local Minima. 2017.
Mei et al. (2018) S. Mei, A. Montanari, and P.-M. Nguyen. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences, 115(33):E7665–E7671, 2018.
Murata (1996) N. Murata. An integral representation of functions using three-layered betworks and their approximation bounds. Neural Networks, 9(6):947–956, 1996.
Nguyen and Hein (2017) Q. Nguyen and M. Hein. The Loss Surface of Deep and Wide Neural Networks. In Proceedings of The 34th International Conference on Machine Learning, volume 70, pages 2603–2612, 2017.
Nitanda and Suzuki (2017) A. Nitanda and T. Suzuki. Stochastic Particle Gradient Descent for Infinite Ensembles. 2017.
Pennington et al. (2018) J. Pennington, S. Schoenholz, and S. Ganguli. The emergence of spectral universality in deep networks. In Proceedings of the 21st International Conference on Artificial Intelligence and Statistics
, volume 84, pages 1924–1932, 2018.
Poole et al. (2016) B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein, and S. Ganguli. Exponential expressivity in deep neural networks through transient chaos. In Advances in Neural Information Processing Systems 29, pages 3360–3368, 2016.
Rahimi and Recht (2008) A. Rahimi and B. Recht. Random Features for Large-Scale Kernel Machines. In J. C. Platt, D. Koller, Y. Singer, and S. T. Roweis, editors, Advances in Neural Information Processing Systems 20, pages 1177–1184. Curran Associates, Inc., 2008.
Rotskoff and Vanden-Eijnden (2018) G. Rotskoff and E. Vanden-Eijnden. Parameters as interacting particles: long time convergence and asymptotic error scaling of neural networks. In Advances in Neural Information Processing Systems 31, pages 7146–7155, 2018.
Sirignano and Spiliopoulos (2020a) J. Sirignano and K. Spiliopoulos. Mean Field Analysis of Neural Networks: A Law of Large Numbers. SIAM Journal on Applied Mathematics, 80(2):725–752, 2020a.
Sirignano and Spiliopoulos (2020b) J. Sirignano and K. Spiliopoulos. Mean field analysis of neural networks: A central limit theorem. Stochastic Processes and their Applications, 130(3):1820–1852, 2020b.
Sonoda and Murata (2014) S. Sonoda and N. Murata. Sampling hidden parameters from oracle distribution. In 24th International Conference on Artificial Neural Networks (ICANN) 2014, volume 8681, pages 539–546, 2014.
Sonoda and Murata (2017) S. Sonoda and N. Murata. Double Continuum Limit of Deep Neural Networks. In ICML 2017 Workshop on Principled Approaches to Deep Learning, pages 1–5, 2017.
Soudry and Carmon (2016) D. Soudry and Y. Carmon. No bad local minima: Data independent training error guarantees for multilayer neural networks. 2016.
Sriperumbudur et al. (2010) B. K. Sriperumbudur, K. Fukumizu, and G. R. G. Lanckriet. Universality, Characteristic Kernels and RKHS Embedding of Measures. Journal of Machine Learning Research, 12(Jul):2389–2410, 2010.
Yehudai and Shamir (2019) G. Yehudai and O. Shamir. On the Power and Limitations of Random Features for Understanding Neural Networks. In Advances in Neural Information Processing Systems 32, pages 6598–6608, 2019.
Yun et al. (2018) C. Yun, S. Sra, and A. Jadbabaie. Global Optimality Conditions for Deep Neural Networks. In International Conference on Learning Representations 2018, pages 1–14, 2018.

Appendix A Proofs

a.1 Theorem 2.3

These formula follow from the computations described in "Reconstruction formula" in Appendix C.2.

a.2 Theorem 2.6

Theorem A.1.

Under Assumption 2.5, the $k_{A}$ is characteristic. If we additionally impose Assumption 2.2 on $σ$ , $k_{A}$ is $c_{0}$ -universal.

Proof.

By direct computation, we have

	$k_{A} (x, y)$	$= \int_{I_{A}^{m} \times T} σ (a \cdot x - b) σ (a \cdot y - b) d a d b$
		$= \int_{I_{A}^{m}} \sum n \in Z \| ˆ σ (n) \|^{2} e^{2 π i n a \cdot (x - y) / T} d a$
		$= \| ˆ σ (0) \|^{2} (2 A)^{m} + \int_{R^{m}} \sum n \in Z ∖ {0} \frac{\| ˆ σ (n) \|^{2}}{\| n / T \|^{m}} 1_{[- n A / T, n A / T]} (a)      =: λ_{A} (a) e^{2 π i a \cdot (x - y)} d a .$

Put $C := | ˆ σ (0) |^{2} (2 A)^{m}$ . Since we assume $# {n | ˆ σ (n) \neq 0} = \infty$ , the support of the function $λ_{A}$ is $R^{m}$ . Therefore, we see that $k_{A} - C$ is universal (see Section 3.2. of (Sriperumbudur et al., 2010)), thus $k_{A}$ is characteristic. Under Assumption 2.2, we have $C = 0$ , and it implies $k_{A}$ itself is $c_{0}$ -universal. ∎

a.3 Theorem 3.2

Here, we define

K ((a, b), (a^{'}, b^{'})) := \int σ (a \cdot x - b) σ (a^{'} \cdot x - b^{'}) d P (x) .

and for $(a, b) \in R^{m} \times T$ , we define

T_{A} [γ] (a, b) := \int_{I_{A}^{m} \times T} γ (a^{'}, b^{'}) K ((a^{'}, b^{'}), (a, b)) d a^{'} d b^{'} .

We define a bounded absolutely integrable function $λ_{A}$ by

λ_{A} (x) := \sum n \neq 0 \frac{| ˆ σ (n) |^{2}}{| n / T |^{m}} 1_{[- n A / T, n A / T]} (x) .

Lemma A.2.

The correspondence $x \mapsto σ_{x}$ is bounded and continuous mapping from $R^{m}$ to $L^{2} (I_{A} \times T, d a d b)$ .

Proof.

We may assume $σ$ is continuous function, thus we immediately see the continuity. The boundedness is obvious. ∎

Corollary A.3.

Let $f \in L^{1} (R^{m}, d x)$ , and let $B$ be a bounded linear operator on $L^{2} (I_{A} \times T, d a d b)$ . Then for any $A^{'} > 0$ , $\int f (x) B [σ_{x}] d x$ is a well-defined elemlent in $L^{2} (I_{A^{'}} \times T, d a d b)$ and satisfy for any $h \in L^{2} (I_{A^{'}}^{m} \times T, d a d b)$ ,

Lemma A.4.

For $g \in L^{1} (R^{m}, d x)$ , we have

{∥ ∥ ∥ \int g (x) T_{A} [σ_{x}] d x ∥ ∥ ∥}_{L^{2} (R^{m} \times T, d a d b)} = {∥ ∥ [g^{♭} λ_{A}]^{♯} q ∥ ∥}_{L^{2} (R^{m}, d x)}^{♭} .

Proof.

Put $ϕ_{A} (x) := λ^{♯} (x)$ . Since

T_{A} [σ_{x}] (a, b) = \int σ_{a, b} (y) q (y) λ_{A}^{♯} (x - y) d y,

for $B > 0$ , by direct computation, we have

	${∥ ∥ ∥ \int g (x) T_{A} [σ_{x}] d x ∥ ∥ ∥}_{A}^{2}$
	$= \int g (x) g (y) q (z) q (w) λ_{B}^{♯} (z - w) λ_{A}^{♯} (x - z) λ_{A}^{♯} (y - w) d x d y d w d z$
	$= \int [g * λ_{A}^{♯}] q (w) [g * λ_{A}^{♯}] q (z) λ_{B}^{♯} (w - z) d w d z$
	$= \int {∣ ∣ ∣ {([g^{♭} λ_{A}]^{♯} q)}_{A}^{♭} (x) ∣ ∣ ∣}^{2} λ_{B} (x) d x .$

By taking $B$ to $\infty$ , we have the formula. ∎

Corollary A.5.

For any $g \in L^{2} (R, d x)$ , the integral $\int g (x) T_{A} [σ_{x}] d x$ is well-defined in the similar manner with the Fourier transform. Moreover, we have

lim A \to \infty {∥ ∥ ∥ \int g (x) T_{A} [σ_{x}] d x - g q ∥ ∥ ∥}_{L^{2} (R^{m} \times T, d a d b)} = 0.

Theorem A.6.

Let $P$ be an absolutely continuous finite Borel measure on $R^{m}$ with density function $q$ . Let $f \in L^{2} (R^{d}, P)$ . Assume $q$ is bounded and $f \in L^{2} (R^{m}, d x)$ . Then we have

γ_{β, A}^{*} [f] = R [\frac{q f}{β + q}] + Δ_{β, A} [f],

(15)

where $Δ_{β, A} [f]$ is an element of $L^{2} (I_{A}^{m} \times T, d a d b)$ such that

lim A \to \infty {∥ ∥ Δ_{β, A} [f] ∥ ∥}_{L^{2} (I_{A}^{m} \times T, d a d b)} = 0.

Proof.

By the theory of the Tikhonov regularization, $γ_{β, A}^{*}$ is explicitly desribed as follows:

γ_{β, A}^{*} [f] = (β + T_{A})^{- 1} R_{A} [f],

where we write $T_{A} := R_{A} S_{A}$ . We denote $\frac{q f}{β + q}$ by $g$ . By direct computation, we have

(β + R_{A} S_{A})^{- 1} R_{A} [f] = R [g] + (β + T_{A})^{- 1} Λ_{A}

where we define $Λ_{A}$ by

	$Λ_{A}$	$:= \int g (q σ_{x} - T_{A} [σ_{x}]) d x$
		$= R [g q] - \int T_{A} [σ_{x}] d x$

By Lemma A.4, we have $Λ_{A} \in L^{2} (R^{m} \times T, d a d b)$ . By Corollary A.5, we see that $λ_{A} \to 0$ in $L^{2} (R^{m} \times T, d a d b)$ . Therefore, we define

Δ_{β, A} [f] := (β + T_{A})^{- 1} Λ_{A}

and the limit of $Δ_{β, A} [f]$ is zero as $A \to \infty$ . ∎

a.4 Theorem 3.3

Here we prove the following statement:

Theorem A.7 (Theorem 3.3).

Let $f \in L^{2} (R^{m}, P)$ . For every $p \in N$ , let $λ_{p} := \frac{(2 A)^{m}}{p} \sum_{i = 1}^{p} δ_{(a_{i}, b_{i})}$ with $(a_{i}, b_{i}) \in I_{A}^{m} \times T$ . Assume that $λ_{p}$ weakly converges to the Lebesgue measure $d a d b$ on $I_{A}^{m} \times T$ . Here, the weak convergence is in the sense that $\int_{I_{A}^{m} \times T} h d λ_{p} (a, b) \to \int_{I_{A}^{m} \times T} h d a d b$ for any bounded continuous function $h$ on $I_{A}^{m} \times T$ . Then the minimimizer $γ_{β, A}^{p} [f]$ of

min γ {∥ f - S_{p} [γ] ∥_{L^{2} (R^{m}, P)}^{2} + β ∥ γ ∥_{L^{2} (I_{A}^{m} \times T, λ_{p})}^{2}}

converges to the minimizer $γ_{β, A}^{*} [f]$ of ${min}_{γ} L_{β, A} (γ; f)$ in the sense $∥ γ_{β, A}^{p} [f] - γ_{β, A}^{*} [f] ∥$

Gradient Descent Converges to Ridgelet Spectrum

Authors

Gradient Descent Converges to Minimizers

Blockwise Adaptivity: Faster Training and Better Generalization in Deep Learning

Continuous vs. Discrete Optimization of Deep Neural Networks

Natural Analysts in Adaptive Data Analysis

A SOM-based Gradient-Free Deep Learning Method with Convergence Analysis

Parameter Norm Growth During Training of Transformers

How to Learn when Data Reacts to Your Model: Performative Gradient Descent

This week in AI

1 Introduction

Main Theorem.

Notation.

2 Ridgelet Transforms on the Torus

2.1 Ridgelet transform

Definition 2.1.

Assumption 2.2 (admissible condition).

Theorem 2.3.

Corollary 2.4.

2.2 Reproducing kernel Hilbert space with inner product of features

Assumption 2.5.

Theorem 2.6.

Corollary 2.7.

Proof.

2.3 Integral representation of neural networks

Definition 2.8.

Proposition 2.9.

Proof.

3 Main Results

3.1 Square loss minimization for the integral representation

Proposition 3.1.

Proof.

3.2 An explicit representation of the global minimizer

Theorem 3.2.

3.3 Relation to the 2-layer finite neural networks

Theorem 3.3.

Corollary 3.4.

4 Numerical Simulation

4.1 Scatter plots of GD trained parameters.

4.2 Ridgelet spectrum

4.3 Results

5 Related Works

6 Conclusion

Broader Impact

Acknowledgments

References

Appendix A Proofs

a.1 Theorem 2.3

a.2 Theorem 2.6

Theorem A.1.

Proof.

a.3 Theorem 3.2

Lemma A.2.

Proof.

Corollary A.3.

Lemma A.4.

Proof.

Corollary A.5.

Theorem A.6.

Proof.

a.4 Theorem 3.3

Theorem A.7 (Theorem 3.3).