Fast learning rate of deep learning via a kernel perspective

1 Introduction

Deep learning has been showing great success in several applications such as computer vision, natural language processing, and many other area related to pattern recognition. Several high-performance methods have been developed and it has been revealed that deep learning possesses great potential. Despite the development of practical methodologies, its theoretical understanding is not satisfactory. Wide rage of researchers including theoreticians and practitioners are expecting deeper understanding of deep learning.

Among theories of deep learning, a well developed topic is its expressive power. It has been theoretically shown that deep neural network has exponentially large expressive power against the number of layers. For example, Montufar et al. (2014) showed that the number of polyhedral regions created by deep neural network can exponentially grow as the number of layers increases. Bianchini and Scarselli (2014)

showed that the Betti numbers of the level set of a function created by deep neural network grows up exponentially against the number of layers. Other researches also concluded similar facts using different notions such as tensor rank and extrinsic curvature

(Cohen et al., 2016; Cohen and Shashua, 2016; Poole et al., 2016).

Another important issue in neural network theories is its universal approximation capability. It is well known that 3-layer neural networks have the ability, and thus the deep neural network also does (Cybenko, 1989; Hornik, 1991; Sonoda and Murata, 2015). When we discuss the universal approximation capability, the target function that is approximated is arbitrary and the theory is highly nonparametric in its nature.

Once we knew the expressive power and universal approximation capability of deep neural network, the next theoretical question naturally arises in its generalization error. The generalization ability is typically analyzed by evaluating the Rademacher complexity. Bartlett (1998) studied 3-layer neural networks and characterized its Rademacher complexity using the norm of weights. Koltchinskii and Panchenko (2002) studied deep neural network and derived its Rademacher complexity under norm constraints. More recently, Neyshabur et al. (2015) analyzed the Rademacher complexity based on more generalized norm, and Sun et al. (2015) derived a generalization error bound with a large margine assumption. As a whole, the studies listed above derived $O (1 / \sqrt{n})$ convergence of the generalization error where $n$ is the sample size. One concern in this line of convergence analyses is that the convergence of the generalization error is only $O (1 / \sqrt{n})$ where $n$

is the sample size. Although this is minimax optimal, it is expected that we could show faster convergence rate with some additional assumptions such as strong convexity of the loss function. Actually, in a regular parametric model, we have

O (1 / n)

convergence of the generalization error (Hartigan et al., 1998). Moreover, the generalization error bound has been mainly given in finite dimensional models. As we have observed, the deep neural network possesses exponential expressive power and universal approximation capability which are highly nonparametric characterizations. This means that the theories are developed separately in the two regimes; finite dimensional parametric model and infinite dimensional nonparametric model. Therefore, theories that connect these two regimes are expected to comprehensively understand statistical performance of deep learning.

In this paper, we consider both of empirical risk minimization and Bayesian deep learning and analyze the generalization error using the terminology of kernel methods. Consequently, (i) we derive a faster learning rate than $O (1 / \sqrt{n})$

and (ii) we connect the finite dimensional regime and the infinite dimensional regime based on the theories of kernel methods. The empirical risk minimization is a typical approach to learn the deep neural network model. It is usually performed by applying stochastic gradient descent with the back-propagation technique

(Widrow and Hoff, 1960; Amari, 1967; Rumelhart et al., ). To avoid over-fitting, such techniques as regularization and dropout have been employed (Srivastava et al., 2014)

. Although the practical techniques for the empirical risk minimization have been extensively studied, there is still much room for improvement in its generalization error analysis. Bayesian deep learning has been recently gathering more attentions mainly because it can deal with the estimation uncertainty in a natural way. Examples of Bayesian deep learning researches include probabilistic backpropagation

(Hernandez-Lobato and Adams, 2015), Bayesian dark knowledge (Balan et al., 2015), weight uncertainty by Bayesian backpropagation (Blundell et al., 2015), dropout as Bayesian approximation (Gal and Ghahramani, 2016). To analyze a sharper generalization error bound, we utilize the so-called local Rademacher complexity technique for the empirical risk minimization method (Mendelson, 2002; Bartlett et al., 2005; Koltchinskii, 2006; Giné and Koltchinskii, 2006), and, as for the Bayesian method, we employ the theoretical techniques developed to analyze nonparametric Bayes methods (Ghosal et al., 2000; van der Vaart and van Zanten, 2008, 2011). These analyses are quite advantageous to the typical Rademacher complexity analysis because we can obtain convergence rate between

O (1 / n)

and

O (1 / \sqrt{n})

which is faster than that of the standard Rademacher complexity analysis

O (1 / \sqrt{n})

. As for the second contribution, we first introduce an integral form of deep neural network as performed in the research of the universal approximation capability of 3-layer neural networks (Sonoda and Murata, 2015). This allows us to have a nonparametric model of deep neural network as a natural extension of usual finite dimensional models. Afterward, we define a reproducing kernel Hilbert space (RKHS) corresponding to each layer like in Bach (2017, 2015). By doing so, we can borrow the terminology developed in the kernel method into the analysis of deep learning. In particular, we define the degree of freedom of the RKHS as a measure of complexity of the RKHS (Caponnetto and de Vito, 2007; Bach, 2015), and based on that, we evaluate how large a finite dimensional model should be to approximate the original infinite dimensional model with a specified precision. These theoretical developments reveal that there appears bias-variance trade-off. That is, there appears trade-off between the size of the finite dimensional model approximating the nonparametric model and the variance of the estimator. We will show that, by balancing the trade-off, a fast convergence rate is derived. In particularly, the optimal learning rate of the kernel method is reproduced from our deep learning analysis due to the fact that the kernel method can be seen as a 3-layer neural network with an infinite dimensional internal layer. A remarkable property of the derived generalization error bound is that the error is characterized by the complexities of the RKHSs defined by the degree of freedom. Moreover, the notion of the degree of freedom gives a practical implication about determination of the width of the internal layers.

The obtained generalization error bound is summarized in Table 1¹¹1 $a \lor b$ indicates $max {a, b}$ ..

	Error bound
General setting	$L \sum_{ℓ = 2}^{L} R^{L - ℓ + 1} λ_{ℓ} + \frac{σ^{2} + {^R}_{\infty}^{2}}{n} \sum_{ℓ = 1}^{L} m_{ℓ} m_{ℓ + 1} log (n)$
	under an assumption that $m_{ℓ} ≳ N_{ℓ} (λ_{ℓ}) log (N_{ℓ} (λ_{ℓ}))$ .
Finite dimensional model	$\frac{σ^{2} + {^R}_{\infty}^{2}}{n} \sum_{ℓ = 1}^{L} m_{ℓ}^{} m_{ℓ + 1}^{} log (n)$
	where $m_{ℓ}^{*}$ is the true width of the $ℓ$ -th internal layer.
Polynomial decay eigenvalue	$L \sum_{ℓ = 2}^{L} (R \lor 1)^{L - ℓ + 1} n^{- \frac{1}{1 + 2 s_{ℓ}}} log (n) + \frac{d_{x}^{2}}{n} log (n)$
	where $s_{ℓ}$ is the decay rate of the eigenvalue of the kernel
	function on the $ℓ$ -th layer.

Table 1: Summary of derived bounds for the generalization error

∥ ˆ f - f^{o} ∥_{L_{2} (P (X))}^{2}

where

n

is the sample size,

R

is the norm of the weight in the internal layers,

{^R}_{\infty}

is an

L_{\infty}

-norm bound of the functions in the model,

σ

is the observation noise,

d_{x}

is the dimension of the input,

m_{ℓ}

is the width of the

ℓ

-th internal layer and

N_{ℓ} (λ_{ℓ})

for (

λ_{ℓ} > 0

) is the degree of freedom (Eq. (5)).

2 Integral representation of deep neural network

Here we give our problem settings and the model that we consider in this paper. Suppose that $n$ input-output observations $D_{n} = (x_{i}, y_{i})_{i = 1}^{n} \subset R^{d_{x}} \times R$ are independently identically generated from a regression model

y_{i} = f^{o} (x_{i}) + ξ_{i} (i = 1, \dots, n)

where $(ξ_{i})_{i = 1}^{n}$ is an i.i.d. sequence of Gaussian noises $N (0, σ^{2})$ with mean 0 and variance $σ^{2}$ , and $(x_{i})_{i = 1}^{n}$ is generated independently identically from a distribution $P (X)$ with a compact support in $R^{d_{x}}$ . The purpose of the deep learning problem we consider in this paper is to estimate $f^{o}$ from the $n$ observations $D_{n}$ .

To analyze the generalization ability of deep learning, we specify a function class in which the true function $f^{o}$ is included, and, by doing so, we characterize the “complexity” of the true function in a correct way.

In order to give a better intuition, we first start from the simplest model, the 3-layer neural network. Let $η$

be a nonlinear activation function such as ReLU

(Nair and Hinton, 2010; Glorot et al., 2011);

η (x) = (max {x_{i}, 0})_{i = 1}^{d}

for a

d

-dimensional vector

x \in R^{d}

. The 3-layer neural network model is represented by

f (x) = W^{(2)} η (W^{(1)} x + b^{(1)}) + b^{(2)}

where we denote by $m_{2}$ the number of nodes in the internal layer, and $W^{(2)} \in R^{1 \times m_{2}}$ , $W^{(1)} \in R^{m_{2} \times d_{x}}$ , $b^{(1)} \in R^{m_{2}}$ and $b^{(2)} \in R$ . It is known that this model is universal approximator and it is important to consider its integral form

f (x) = \int h (w, b) η (w^{⊤} x + b) d w d b + b^{(2)} .

(1)

where $(w, b) \in R^{d_{x}} \times R$ is a hidden parameter, $h : R^{d_{x}} \times R \to R$ is a function version of the weight matrix $W^{(2)}$ , and $b^{(2)} \in R$ is the bias term. This integral form appears in many places to analyze the capacity of the neural network. In particular, through the ridgelet analysis, it is shown that there exists the integral form corresponding to any $f \in L_{1} (R^{d_{x}})$

which has an integrable Fourier transform for an appropriately chosen activation function

η

such as ReLU (Sonoda and Murata, 2015).

Motivated by the integral form of the 3-layer neural network, we consider a more general representation for deeper neural network. To do so, we define a feature space on the $ℓ$

-th layer. The feature space is a a probability space

(T_{ℓ}, B_{ℓ}, Q_{ℓ})

where

T_{ℓ}

is a Polish space,

B_{ℓ}

is its Borel algebra, and

Q_{ℓ}

is a probability measure on

(T_{ℓ}, B_{ℓ})

. This is introduced to represent a general (possibly) continuous set of features as well as a discrete set of features. For example, if the

ℓ

-th internal layer is endowed with a

d_{ℓ}

-dimensional finite feature space, then

T_{ℓ} = {1, \dots, d_{ℓ}}

. On the other hand, the integral form (1) corresponds to a continuous feature space

T_{2} = {(w, b) \in R^{d_{x}} \times R}

in the second layer. Now the input

x

is a

d_{x}

-dimensional real vector, and thus we may set

T_{1} = {1, \dots, d_{x}}

. Since the output is one dimensional, the output layer is just a singleton

T_{L + 1} = {1}

. Based on these feature spaces, our integral form of the deep neural network is constructed by stacking the map on the

ℓ

-th layer

f_{ℓ}^{o} : L_{2} (Q_{ℓ}) \to L_{2} (Q_{ℓ + 1})

given as


	$f_{ℓ}^{o} [g] (τ) = \int_{T_{ℓ}} h_{ℓ}^{o} (τ, w) η (g (w)) d Q_{ℓ} (w) + b_{ℓ}^{o} (τ),$	(2a)
where $h_{ℓ}^{o} (τ, w)$ corresponds to the weight of the feature $w$ for the output $τ$ and $h_{ℓ}^{o} \in L_{2} (Q_{ℓ + 1} \times Q_{ℓ})$ and $h_{ℓ}^{o} (τ, \cdot) \in L_{2} (Q_{ℓ + 1})$ for all $τ \in T_{ℓ + 1}$ ²²2 Note that, for $g \in L_{2} (Q_{ℓ})$ , $f_{ℓ} [g]$ is also square integrable with respect to $L_{2} (Q_{ℓ + 1})$ if $η$ is Lipschitz continuous because $h \in L_{2} (Q_{ℓ + 1} \times Q_{ℓ})$ .. Specifically, the first and the last layers are represented as

	$f_{1}^{o} [x] (τ) = d_{x} \sum j = 1 h_{1}^{o} (τ, j) x_{j} Q_{1} (j) + b_{1}^{o} (τ),$	(2b)
	$f_{L}^{o} [g] (1) = \int_{T_{L}} h_{L}^{o} (w) η (g (w)) d Q_{L} (w) + b_{L}^{o},$	(2c)

where we wrote $h_{L}^{o} (w)$ to indicate $h_{L}^{o} (1, w)$ for simplicity because $T_{L + 1} = {1}$ . Then the true function $f^{o}$ is given as

f^{o} (x) = f_{L}^{o} \circ f_{L - 1}^{o} \circ \dots \circ f_{1}^{o} (x) .

(3)

Since, the shallow 3-layer neural network is a universal approximator, and so is our generalized deep neural network model (3). It is known that deep neural network tends to give more efficient representation of a function than the shallow network. Actually, Eldan and Shamir (2016) gave an example of a function that the 3-layer neural network cannot approximate under a precision unless its with is exponential in the input dimension but the 4-layer neural network can approximate with polynomial order widths (see Safran and Shamir (2016) for other examples). In other words, each layer of a deep neural network can be much “simpler” than one of a shallow network (more rigorous definition of complexity of each layer will be given in the next section). Therefore, it is quite important to consider the integral representation of a deep neural network rather than a 3-layer network.

The integral representation is natural also from the practical point of view. Indeed, it is well known that the deep neural network learns a simple pattern in the early layers and it gradually extracts more complicated features as the layer is going up. The trained feature is usually continuous one. For example, in computer vision tasks, the second layer typically extracts gradients toward several degree angles (Krizhevsky et al., 2012). The angle is a continuous variable and thus the feature space should be continuous to cover all angles. On the other hand, the real network discretize the feature space because of limitation of computational resources. Our theory introduced in the next section offers a measure to evaluate this discretization error.

3 Finite approximation of the integral form

The integral form is a convenient way to describe the true function. However, it is not useful to estimate the function. When we estimate that, we need to discretize the integrals by finite sums due to limitation of computational resources as we do in practice. In other word, we consider the usual finite sum deep learning model as an approximation of the integral form. However, the discrete approximation induces approximation error. Here we give an upper bound of the approximation error. Naturally, there arises the notion of bias and variance trade-off, that is, as the complexity of the finite model increases the “bias” (approximation error) decreases but the “variance” for finding the best parameter in the model increases. Afterwards, we will bound the variance for estimating the finite approximation in Section 4.3. Combining these two notions, it is possible to quantify the bias-variance trade-off and find the best strategy to minimize the entire generalization error.

The approximation error analysis of the deep neural network can be well executed by utilizing notions of the kernel method. Here we construct RKHS for each layer in a way analogous to Bach (2015, 2017) who studied shallow learning and the kernel quadrature rule. Let the output of the $ℓ$ -th layer be $F_{ℓ}^{o} (x, τ) := (f_{ℓ}^{o} \circ \dots \circ f_{1}^{o} (x)) (τ) .$ We define a reproducing kernel Hilbert space (RKHS) corresponding to the $ℓ$ -th layer ( $ℓ \geq 2$ ) by introducing its associated kernel function $k_{ℓ} : R^{d_{x}} \times R^{d_{x}} \to R$ . We define the positive definite kernel $k_{ℓ}$ as

k_{ℓ} (x, x^{'}) := \int_{T_{ℓ}} η (F_{ℓ - 1}^{o} (x, τ)) η (F_{ℓ - 1}^{o} (x^{'}, τ)) d Q_{ℓ} (τ) .

It is easy to check that $k_{ℓ}$ is actually symmetric and positive definite. It is known that there exists a unique RKHS $H_{ℓ}$ corresponding the kernel $k_{ℓ}$ (Aronszajn, 1950). Close investigation of the RKHS for several examples for shallow network has been given in (Bach, 2017).

Under this setting, all arguments at the $ℓ$ -th layer can be carried out through the theories of kernel methods. Importantly, for $g \in H_{ℓ}$ , there exists $h \in L_{2} (Q_{ℓ})$ such that

g (x) = \int_{T_{ℓ}} h (τ) η (F_{ℓ - 1}^{o} (x, τ)) d Q_{ℓ} (τ) .

Moreover, the norms of $g$ and $h$ are connected as

∥ g ∥_{H_{ℓ}} = ∥ h ∥_{L_{2} (Q_{ℓ})},

(4)

(Bach, 2015, 2017). Therefore, the function

x \mapsto \int_{T_{ℓ}} h_{ℓ}^{o} (τ, w) η (F_{ℓ - 1}^{o} (x, w)) d Q_{ℓ} (w),

representing the magnitude of a feature $τ \in T_{ℓ + 1}$ for the input $x$ is included in the RKHS and its RKHS norm is equivalent to that of the internal layer weight $∥ h^{o} (τ, \cdot) ∥_{L_{2} (Q_{ℓ})}$ because of Eq. (4).

To derive the approximation error, we need to evaluate the “complexity” of the RKHS. Basically, the complexity of the $ℓ$ -th layer RKHS $H_{ℓ}$ is controlled by the behavior of the eigenvalues of the kernel. To formally state this notion, we introduce the integral operator associated with the kernel $k_{ℓ}$ defined as

	$T_{ℓ} :$	$g \mapsto \int_{X} k_{ℓ} (\cdot, x) g (x) d P (x),$
		$L_{2} (P (X)) \to L_{2} (P (X)) .$

If the kernel function admits an orthogonal decomposition

k_{ℓ} (x, x^{'}) = \infty \sum j = 1 μ_{j}^{(ℓ)} ϕ_{j}^{(ℓ)} (x) ϕ_{j}^{(ℓ)} (x^{'}),

in $L_{2} (P (X) \times P (X))$ where $(μ_{j}^{(ℓ)})_{j = 1}^{\infty}$ is the sequence of the eigenvalues ordered in decreasing order, and $(ϕ_{j}^{(ℓ)})_{j = 1}^{\infty}$ forms an orthonormal system in $L_{2} (P (X))$ , then for $g (x) = \sum_{j = 1}^{\infty} α_{j} ϕ_{j}^{(ℓ)} (x)$ , the integral operation is expressed as $T_{ℓ} g = \sum_{j = 1}^{\infty} α_{j} μ_{j}^{(ℓ)} ϕ_{j}^{(ℓ)}$ (see Steinwart and Christmann (2008); Steinwart and Scovel (2012) for more details). Therefore each eigenvalue $μ_{j}^{(ℓ)}$ plays a role like a “filter” for each component $ϕ_{j}^{(ℓ)}$ . Here it is known that for all $g \in H_{ℓ}$ , there exists $¯ h \in L_{2} (P (X))$ such that $g = T_{ℓ} ¯ h$ and $∥ g ∥_{H_{ℓ}} = ∥ ¯ h ∥_{L_{2} (P (X))}$ (Caponnetto and de Vito, 2007; Steinwart et al., 2009). Combining this with Eq. (4), we have $∥ g ∥_{H_{ℓ}} = ∥ h ∥_{L_{2} (Q_{ℓ})} = ∥ ¯ h ∥_{L_{2} (P (X))}$

Based on the integral operator $T_{ℓ}$ , we define the degree of freedom $N_{ℓ} (λ)$ of the RKHS as

N_{ℓ} (λ) = T r [(T_{ℓ} + λ)^{- 1} T_{ℓ}]

(5)

for $λ > 0$ . The degree of freedom can be represented as $N_{ℓ} (λ) = \sum_{j = 1}^{\infty} \frac{μ_{j}^{(ℓ)}}{μ_{j}^{(ℓ)} + λ}$ by using the eigenvalues of the kernel.

Now, we assume that the true function $f^{o}$ satisfies a norm condition as follows.

Assumption 1

For each $ℓ$ , $h_{ℓ}^{o}$ and $b_{ℓ}^{o}$ satisfy that

	$∥ h_{ℓ}^{o} (τ, \cdot) ∥_{L_{2} (Q_{ℓ})} \leq R (\forall τ \in T_{ℓ}),$
	$\| b_{ℓ}^{o} (τ) \| \leq R_{b} (\forall τ \in T_{ℓ}) .$

By Eq. (4), the first assumption $∥ h_{ℓ}^{o} (τ, \cdot) ∥_{L_{2} (Q_{ℓ})} \leq R$ is interpreted as $F_{ℓ}^{o} (τ, \cdot) \in H_{ℓ}$ and $∥ F_{ℓ}^{o} (τ, \cdot) ∥_{H_{ℓ}} \leq R$ . This means that the feature map $F_{ℓ}^{o} (τ, \cdot)$ in each internal layer is well regulated by the RKHS norm.

Moreover, we also assume that the activation function is scale invariant.

Assumption 2

We assume the following conditions on the activation function $η$ .

$η$ is scale invariant: $η (a x) = a η (x)$ for all $a > 0$ and $x \in R^{d}$ (for arbitrary $d$ ).
$η$ is 1-Lipschitz continuous: $| η (x) - η (x^{'}) | \leq ∥ x - x^{'} ∥$ for all $x, x^{'} \in R^{d}$ .

The first assumption on the scale invariance is essential to derive tight error bounds. The second one ensures that deviation in each layer does not affect the output so much. The most important example of an activation function that satisfies these conditions is ReLU activation. Another one is the identity map $η (x) = x$ .

Finally we assume that the input distribution has a compact support.

Assumption 3

The support of $P (X)$ is compact and it is bounded as

∥ x ∥_{\infty} := max 1 \leq i \leq d_{x} | x_{i} | \leq D_{x} (\forall x \in s u p p (P (X))) .

We consider a finite dimensional approximation $f^{*}$ given as follows: let $m_{ℓ}$ be the number of nodes in the $ℓ$ -th internal layer (we set the dimensions of the output and input layers to $m_{L + 1} = 1$ and $m_{1} = d_{x}$ ) and consider a model

	$f_{ℓ}^{*} (g) = W^{(ℓ)} η (g) + b^{(ℓ)} (g \in R^{m_{ℓ}}, ℓ = 2, \dots, L),$
	$f_{1}^{*} (x) = W^{(1)} x + b^{(1)},$
	$f^{} (x) = f_{L}^{} \circ f_{L - 1}^{} \circ \dots \circ f_{1}^{} (x),$

where $W^{(ℓ)} \in R^{m_{ℓ + 1} \times m_{ℓ}}$ and $b^{(ℓ)} \in R^{m_{ℓ + 1}}$ .

Theorem 1 (Finite approximation error bound of the nonparametric model)

For any $1 > δ > 0$ and $λ_{ℓ} > 0$ , suppose that

m_{ℓ} \geq 5 N_{ℓ} (λ_{ℓ}) log (32 N_{ℓ} (λ_{ℓ}) / δ) (ℓ = 2, \dots, L),

then there exist $W^{(ℓ)} \in R^{m_{ℓ + 1} \times m_{ℓ}}$ and $b^{(ℓ)} \in R^{m_{ℓ + 1}}$ such that, by letting ${^c}_{δ} = \frac{4}{1 - δ}$ ,


	$∥ W^{(ℓ)} ∥_{F}^{2} \leq {^c}_{δ} R^{2} (ℓ = 1, \dots, L),$		(6a)
	$∥ b^{(ℓ)} ∥ \leq R_{b} / (1 - δ) (ℓ = 1, \dots, L),$		(6b)

and

	$∥ f^{o} - f^{*} ∥_{L_{2} (P (X))} \leq L \sum ℓ = 2 2 \sqrt{{^c}_{δ}^{L - ℓ}} R^{L - ℓ + 1} \sqrt{λ_{ℓ}},$		(7)
	$∥ f^{*} ∥_{\infty} \leq (\sqrt{{^c}_{δ}} R)^{L} D_{x} + L \sum ℓ = 1 (\sqrt{{^c}_{δ}} R)^{L - ℓ} \frac{R_{b}}{1 - δ} .$		(8)

The proof is given in Appendix A. This theorem is proven by borrowing the theoretical technique recently developed for the kernel quadrature rule (Bach, 2015). We also employed some techniques analogous to the analysis of the low rank tensor estimation (Suzuki, 2015; Kanagawa et al., 2016; Suzuki et al., 2016). Intuitively, the degree of freedom $N_{ℓ} (λ_{ℓ})$ is the intrinsic dimension of the $ℓ$ -th layer to achieve the $\sqrt{λ_{ℓ}}$ approximation error. Indeed, we show in the proof that the $ℓ$ -th layer is approximated by the $m_{ℓ}$ dimensional nodes with the precision $\sqrt{λ_{ℓ}}$ under the condition $m_{ℓ} = Ω (N_{ℓ} (λ_{ℓ}) log (N_{ℓ} (λ_{ℓ})))$ . The error bound (7) indicates that the total approximation error of the whole network is basically obtained by summing up the approximation error $\sqrt{λ_{ℓ}}$ of each layer where the factor $\sqrt{{^c}_{δ}^{L - ℓ}} R^{L - ℓ + 1}$ is a Lipschitz constant for error propagation.

We would like to emphasize that the approximation error bound (7) and the norm bounds (6) of $W^{(ℓ)}$ and $b^{(ℓ)}$ are independent of the dimensions $(m_{ℓ})_{ℓ = 1}^{L}$ of the internal layers. This is due to the scale invariance property of the activation function. This is quite beneficial to derive a tight generalization error bound. Indeed, without the scale invariance, we only have a much looser bound $∥ f^{o} - f^{*} ∥_{L_{2} (P (X))} \leq \sum_{ℓ = 2}^{L} 2 \sqrt{m_{ℓ + 1} {^c}_{δ}^{L - ℓ}} R^{L - ℓ + 1} \sqrt{λ_{ℓ}}$ , and $∥ W^{(ℓ)} ∥_{F}^{2} \leq m_{ℓ + 1} {^c}_{δ} R^{2}$ , $∥ b^{(ℓ)} ∥^{2} \leq m_{ℓ + 1} R_{b}^{2}$ which depend on the dimensions $(m_{ℓ})_{ℓ = 1}^{L}$ and could be huge for small $λ_{ℓ}$ . This would support the practical success of using the ReLU activation.

Let the norm bounds shown in Theorem 1 be

¯ R = \sqrt{{^c}_{δ}} R, {¯ R}_{b} = R_{b} / (1 - δ) .

Remind that Theorem 1 gives an upper bound of the infinity norm of $f^{*}$ , that is, $∥ f^{*} ∥_{\infty} \leq {^R}_{\infty}$ where

{^R}_{\infty} = {¯ R}^{L} D_{x} + L \sum ℓ = 1 {¯ R}^{L - ℓ} {¯ R}_{b} .

Let the set of finite dimensional functions with the norm constraint (6) be

F = {f (x) = (W^{(L)} η (\cdot) + b^{(L)}) \circ \dots \circ (W^{(1)} x + b^{(1)}) ∣ ∥ W^{(ℓ)} ∥_{F} \leq ¯ R, ∥ b^{(ℓ)} ∥ \leq {¯ R}_{b} (ℓ = 1, \dots, L)} .

Then, we can show that the infinity norm of $F$ is also uniformly bounded as the following lemma.

Lemma 1

For all $f \in F$ , it holds that $∥ f ∥_{\infty} \leq {^R}_{\infty} .$

The proof is given in Appendix A.2. Because of this, we can derive the generalization error bound with respect to the population $L_{2}$ -norm instead of the empirical $L_{2}$ -norm. One can check that $∥ f ∥_{\infty} \leq {^R}_{\infty}$ for all $f \in F$ by Lemma 1 or Lemma 3.

4 Generalization error bounds

In this section, we define the two estimators in the finite dimensional model introduced in the last section: the empirical risk minimizer and the Bayes estimator. The generalization error bounds for both of these estimators are derived. We also give some examples in which the generalization error is analyzed in details.

4.1 Notations

Before we state the generalization error bounds, we prepare some notations. Let $^G = L {¯ R}^{L - 1} D_{x} + \sum_{ℓ = 1}^{L} {¯ R}^{L - ℓ},$ and define ${^δ}_{1, n}$ , ${^δ}_{2, n}$ as³³3We define ${log}_{+} (x) = max {1, log (x)}$ .

	${^δ}_{1, n}$	$= L \sum ℓ = 2 2 \sqrt{{^c}_{δ}^{L - ℓ}} R^{L - ℓ + 1} \sqrt{λ_{ℓ}},$
	${^δ}_{2, n}^{2}$	$=2nL∑ℓ=1mℓmℓ+1log+(1+4√2^Gmax{¯R,¯Rb}√nσ√∑Lℓ=1mℓmℓ+1).$

Note that ${^δ}_{1, n}$ is the finite approximation error given in Theorem 1. Roughly speaking, ${^δ}_{2, n}$ corresponds to the amount of deviation of the estimators in the finite dimensional model.

4.2 Empirical risk minimization

In this section, we define the empirical risk minimizer and investigate its generalization error. Let the empirical risk minimizer be $ˆ f$ :

ˆ f := a r g m i n f \in F n \sum i = 1 (y_{i} - f (x_{i}))^{2} .

Note that there exists at least one minimizer because the parameter set corresponding to $F$ is a compact set and $η$ is a continuous function. $ˆ f$ needs not necessarily be the exact minimizer but it could be an approximated minimizer. We, however, assume $ˆ f$ is the exact minimizer for theoretical simplicity. In practice, the empirical risk minimizer is obtained by using the back-propagation technique. The regularization for the norm of the weight matrices and the bias terms are implemented by using the $L_{2}$ -regularization and the drop-out techniques.

The generalization error of the empirical risk minimizer is bounded as in the following theorem.

Theorem 2

For any $δ > 0$ and $λ_{ℓ} > 0$ , suppose that

m_{ℓ} \geq 5 N_{ℓ} (λ_{ℓ}) log (32 N_{ℓ} (λ_{ℓ}) / δ) (ℓ = 2, \dots, L) .

(9)

Then, there exists universal constants $C_{1}$ such that, for any $r > 0$ and $~ r > 1$ ,

∥ ˆ f - f^{o} ∥_{L_{2} (P (X))}^{2} \leq

C3{~r^δ21,n+(σ2+^R2∞)^δ22,n+(^R2∞+σ2)n[log+(√nmin{σ/^R∞,1})+r]}

with probability $1 - exp (- \frac{n {^δ}_{1, n}^{2} (~ r - 1)^{2}}{11 {^R}_{\infty}^{2}}) - 2 exp (- r)$ for every $r > 0$ and ${~ r}^{'} > 1$ .

The proof is given in Appendix C. This theorem can be shown by evaluating the covering number of the model $F$ and applying the local Rademacher complexity technique (Mendelson, 2002; Bartlett et al., 2005; Koltchinskii, 2006; Giné and Koltchinskii, 2006).

It is easily checked that the third term of the right side ( $(^R2∞+σ2)n[log+(√nmin{σ/^R∞,1})+r]$ ) is smaller than the first two terms, therefore the generalization error bound can be simply evaluated as

∥ ˆ f - f^{o} ∥_{L_{2}}^{2} = O_{p} ({^δ}_{1, n}^{2} + {^δ}_{2, n}^{2}) .

Based on a rough evaluation

{^δ}_{1, n}^{2} ≃ L L \sum ℓ = 1 λ_{ℓ}, {^δ}_{2, n}^{2} ≃ L \sum ℓ = 1 \frac{m_{ℓ} m_{ℓ + 1}}{n} log (n),

and the constraint $m_{ℓ} ≳ N_{ℓ} (λ_{ℓ}) log (N_{ℓ} (λ_{ℓ}))$ , we can observe the bias-variance trade-off for the generalization error bound because, as $λ_{ℓ}$ decreases, the required width of the internal layer $m_{ℓ}$ increases by the condition (9) and thus the deviation ${^δ}_{2, n}$ in the finite dimensional model should increase. In other words, if we want to construct a finite dimensional model which well approximates the true function, then a more complicated model is required and we should pay larger variance of the estimator. A key notion for the bias-variance trade-off is the degree of freedom $N_{ℓ} (λ_{ℓ})$ which expresses the “complexity” of the RKHS $H_{ℓ}$ in each layer. The degree of freedom of a complicated RKHS grows up faster than a simpler one as $λ$ goes to 0. This is also informative in practice because, to determine the width of each layer, the degree of freedom gives a good guidance. That is, if the degree of freedom is small compared with the sample size, then we may increase the width of the layer. An estimate of the degree of freedom can be computed from the trained network by computing the Gram matrix corresponding to the kernel induced from the trained network (where the kernel is defined by the finite sum instead of the integral form) and using the eigenvalue of the Gram matrix.

To obtain the best generalization error bound, $(λ_{ℓ})_{ℓ = 1}^{L}$ should be tuned to balance the bias-variance terms (and accordingly $(m_{ℓ})_{ℓ = 2}^{L}$ should also be fine-tuned). The examples of the best achievable generalization error will be shown in Section 4.4.

4.3 Bayes estimator

In this section, we formulate a Bayes estimator and derive its generalization error. To define the Bayes estimator, we just need to specify the prior distribution. Let $B_{d} (C)$ be the ball in the Euclidean space $R^{d}$ with radius $C > 0$ $(B_{d} (C) = {x \in R^{d} ∣ ∥ x ∥ \leq C})$ , and $U (B_{d} (C))$

be the uniform distribution on the ball

B_{d} (C)

. Since Theorem 1 ensures the norms of

W^{(ℓ)}

and

b^{(ℓ)}

are bounded above by

¯ R

and

{¯ R}_{b}

, it is natural to employ a prior distribution that possesses its support on the set of parameters with norms not greater than those norm bounds. Based on this observation, we employ uniform distributions on balls with the radii indicated above as a prior distribution:

W^{(ℓ)} \sim U (B_{m_{ℓ + 1} \times m_{ℓ}} (¯ R)), b^{(ℓ)} \sim U (B_{m_{ℓ + 1}} ({¯ R}_{b})) .

In practice, the Gaussian distribution is also employed instead of the uniform distribution. However, the Gaussian distribution does not give good tail probability bound for the infinity norm of the deep neural network model. That is crucial to develop the generalization error bound. For this reason, we decided to analyze the uniform prior distribution.

The prior distribution on the parameters $(W^{(ℓ)}, b^{(ℓ)})_{ℓ = 1}^{L}$ induces the distribution of the function $f$ in the space of continuous functions endowed with the Borel algebra corresponding to the $L_{\infty} (R^{d_{x}})$ -norm. We denote by $Π$ the induced distribution. Using the prior, the posterior distribution is defined via the Bayes principle:

Π (d f | D_{n}) = \frac{exp (- \sum_{i = 1}^{n} \frac{(y_{i} - f (x_{i}))^{2}}{2 σ^{2}}) Π (d f)}{\int exp (- \sum_{i = 1}^{n} \frac{(y_{i} - f^{'} (x_{i}))^{2}}{2 σ^{2}}) Π (d f^{'})} .

Since the purpose of this paper is to give a theoretical analysis for the generalization error, we do not pursue the computational issue of the Bayesian deep learning. See, for example, Hernandez-Lobato and Adams (2015); Blundell et al. (2015) for practical algorithms.

The following theorem gives how fast the Bayes posterior contracts around the true function.

Theorem 3

Fix arbitrary $δ > 0$ and $λ_{ℓ} > 0 (ℓ = 1, \dots, L)$ , and suppose that the condition (9) on $m_{ℓ}$ is satisfied. Then, for all $r \geq 1$ , the posterior tail probability can be bounded as

	$EDn[Π(f:∥f−fo∥L2(P(X))≥(^δ1,n+σ^δ2,n)r√max{12,33^R2∞σ2}\|Dn)]$
	$\leq exp [- n {^δ}_{1, n}^{2} \frac{(r^{2} - 1)^{2}}{11^R_{\infty}^{2}}] + 12 exp (- n ({^δ}_{1, n} + σ {^δ}_{2, n})^{2} \frac{r^{2}}{8 σ^{2}}) .$

The proof is given in Appendix B. The proof is accomplished by using the technique for non-parametric Bayes methods (Ghosal et al., 2000; van der Vaart and van Zanten, 2008, 2011). Roughly speaking this theorem indicates that the posterior distribution concentrates in the distance ${^δ}_{1, n} + σ {^δ}_{2, n}$ from the true function $f^{o}$ . The tail probability is sub-Gaussian and thus the posterior mass outside the distance ${^δ}_{1, n} + σ {^δ}_{2, n}$ from the true function rapidly decrease. Here we again observe that there appears bias-variance trade-off between ${^δ}_{1, n}$ and ${^δ}_{2, n}$ . This can be understood essentially in the same way as the empirical risk minimization.

From the posterior contraction rate, we can derive the generalization error bound of the posterior mean.

Corollary 1

Under the same setting as in Theorem 3, there exists a universal constant $C_{1}$ such that the generalization error of the posterior mean $ˆ f$ is bounded as

EDn[∥ˆf−fo∥2L2(P(X))]≤C1max{12,33^R2∞σ2}⎡⎢ ⎢⎣⎛⎜ ⎜⎝1+^R∞√n^δ21,n⎞⎟ ⎟⎠(^δ21,n+σ2^δ22,n)+σ2n⎤⎥ ⎥⎦.

Therefore, for sufficiently large $n$ such that $n {^δ}_{1, n}^{2} / {^R}_{\infty}^{2} \geq 1$ (which is the regime of our interest), the generalization error is simply bounded as

∥ˆf−fo∥2L2(P(X))=Op(max{1,^R2∞σ2}(^δ21,n+σ2^δ22,n)).

4.4 Examples

Here, we give some examples of the generalization error bound. We have seen that both of the empirical risk minimizer and the Bayes estimators have a simplified generalization error bound as

∥ ˆ f - f^{o} ∥_{L_{2} (P (X))}^{2} = O_{p} ({^δ}_{1, n}^{2} + {^δ}_{2, n}^{2}) = O_{p} (L L \sum ℓ = 2 {¯ R}^{L - ℓ + 1} λ_{ℓ} + L \sum ℓ = 1 \frac{m_{ℓ} m_{ℓ + 1}}{n} log (n)),

by supposing $σ$ , ${^R}_{\infty}$ and $\sqrt{{^c}_{δ}^{L}} R^{L}$ are in constant order. We evaluate the bound under the best choice of $m_{ℓ}$ balancing the bias-variance trade-off.

One way to balance the terms is to set $λ_{ℓ}$ so that

L \sum ℓ = 2 λ_{ℓ} = L \sum ℓ = 1 \frac{m_{ℓ} m_{ℓ + 1}}{n}

where the $log (n)$ -factor and $L$

are dropped for simplicity. Since the inequality of arithmetic sum geometric mean gives

\sum_{ℓ = 1}^{L} \frac{m_{ℓ} m_{ℓ + 1}}{n} \leq \sum_{ℓ = 1}^{L + 1} \frac{m_{ℓ}^{2}}{n}

, we may set

m_{ℓ}

to satisfy

λ_{ℓ} = \frac{m_{ℓ}^{2}}{n} (ℓ = 2, \dots, L) .

(10)

Considering this relation and the constraint $m_{ℓ} ≳ N_{ℓ} (λ_{ℓ}) log (N_{ℓ})$ (Eq. (9)), we can estimate the best width $m_{ℓ}$ that minimizes the upper bound of the generalization error.

4.4.1 Finite dimensional internal layer

If all RKHSs are finite dimensional, say $m_{ℓ}^{*}$ -dimensional. Then $N_{ℓ} (λ) \leq m_{ℓ}^{*}$ for all $λ \geq 0$ . Therefore, by setting $λ_{ℓ} = 0 (\forall ℓ)$ , the generalization error bound is obtained as

∥ ˆ f - f^{o} ∥_{L_{2} (P (X))}^{2} ≲ \frac{σ^{2} + {^R}_{\infty}^{2}}{n} L \sum ℓ = 1 m_{ℓ}^{*} m_{ℓ + 1}^{*} log (n),

(11)

where we omitted the factors depending only on $log (¯ R {¯ R}_{b}^G)$ . Note that, although there appears the $L_{\infty}$ -norm bound ${^R}_{\infty}$ , this convergence rate is independent of the Lipschitz constant ${¯ R}^{L - ℓ + 1}$ and ${¯ R}_{b}$ up to $log$ -order but is solely dependent on the number of parameters. Moreover, the convergence rate is $O (log (n) / n)$ in terms of the sample size $n$ . This is much faster than the existing bounds that utilize the Rademacher complexity because their bounds are $O (1 / \sqrt{n})$ . This result matches more precise arguments for a finite dimensional 3-layer neural network based on asymptotic expansions (Fukumizu, 1999; Watanabe, 2001) which also showed the generalization error of the 3-layer neural network can be evaluated as $O ((m_{1}^{*} m_{2}^{*} + m_{2}^{*} m_{3}^{*}) / n)$ .

4.4.2 Polynomial decreasing rate of eigenvalues

We assume that the eigenvalue $μ_{j}^{(ℓ)}$ decays in polynomial order as

μ_{j}^{(ℓ)} \leq a_{ℓ} j^{- \frac{1}{s_{ℓ}}},

(12)

for a positive real $0 < s_{ℓ} < 1$ and $a_{ℓ} > 0$ . This is a standard assumption in the analysis of kernel methods (Caponnetto and de Vito, 2007; Steinwart and Christmann, 2008), and it is known that this assumption is equivalent to the usual covering number assumption (Steinwart et al., 2009). For small $s_{ℓ}$ , the decay rate is fast and it is easy to approximate the kernel by another one corresponding to a finite dimensional subspace. Therefore small $s_{ℓ}$ corresponds to a simple model and large $s_{ℓ}$ corresponds to a complicated model. In this setting, the degree of freedom is evaluated as

N_{ℓ} (λ_{ℓ}) ≲ {(λ_{ℓ} / a_{ℓ})}_{ℓ}^{- s_{ℓ}} .

(13)

This can be shown as follows: for any positive integer $M$ , the degree of freedom can be bounded as

	$N_{ℓ} (λ_{ℓ})$	$= \infty \sum j = 1 \frac{μ_{j}^{(ℓ)}}{μ_{j}^{(ℓ)} + λ_{ℓ}} \leq M + \infty \sum j = M + 1 \frac{μ_{j}^{(ℓ)}}{λ_{ℓ}} \leq M + \frac{a_{ℓ}}{λ_{ℓ}} \int_{M}^{\infty} x^{- 1 / s_{ℓ}} d x$
		$\leq M + (a_{ℓ} / λ_{ℓ}) (1 - 1 / s_{ℓ})^{- 1} M^{1 - 1 / s_{ℓ}} .$

Letting $M = {⌈ (a_{ℓ} / λ_{ℓ}) (1 - 1 / s_{ℓ})^{- 1} ⌉}_{ℓ}^{s_{ℓ}}$ to balance the first and the second term, we obatain the evaluation (13). Hence, we can show that, according to Eq. (10),

λ_{ℓ} = a_{ℓ}^{\frac{2 s_{ℓ}}{1 + 2 s_{ℓ}}} n^{- \frac{1}{1 + 2 s_{ℓ}}}

gives the optimal rate, and we obtain the generalization error bound as

∥ ˆ f - f^{o} ∥_{L_{2} (P (X))}^{2} ≲ L L \sum ℓ = 2 (¯ R \lor 1)^{2 (L - ℓ + 1)} a_{ℓ}^{\frac{2 s_{ℓ}}{1 + 2 s_{ℓ}}} n^{- \frac{1}{1 + 2 s_{ℓ}}} log (n) + \frac{d_{x}^{2}}{n} log (n),

(14)

where we omitted the factors depending on $s_{ℓ}, log (¯ R {¯ R}_{b}^G)$ , $σ^{2}$ and ${^R}_{\infty}$ . This indicates that the complexity $s_{ℓ}$ of the RKHS affects the convergence rate directly. As expected, if the RKHSs are simple (that is, $(s_{ℓ})$

Fast learning rate of deep learning via a kernel perspective

Authors

Generalization bound of globally optimal non-convex neural network training: Transportation map estimation by infinite dimensional Langevin dynamics

Autoencoding any Data through Kernel Autoencoders

An Online Projection Estimator for Nonparametric Regression in Reproducing Kernel Hilbert Spaces

The Kolmogorov Infinite Dimensional Equation in a Hilbert space Via Deep Learning Methods

How much is optimal reinsurance degraded by error?

Fast Learning Rate of Non-Sparse Multiple Kernel Learning and Optimal Regularization Strategies

Sub-linear convergence of a stochastic proximal iteration method in Hilbert space

1 Introduction

2 Integral representation of deep neural network

3 Finite approximation of the integral form

Assumption 1

Assumption 2

Assumption 3

Theorem 1 (Finite approximation error bound of the nonparametric model)

Lemma 1

4 Generalization error bounds

4.1 Notations

4.2 Empirical risk minimization

Theorem 2

4.3 Bayes estimator

Theorem 3

Corollary 1

4.4 Examples

4.4.1 Finite dimensional internal layer

4.4.2 Polynomial decreasing rate of eigenvalues

Fast learning rate of deep learning via a kernel perspective

Authors

Related Research

Generalization bound of globally optimal non-convex neural network training: Transportation map estimation by infinite dimensional Langevin dynamics

Autoencoding any Data through Kernel Autoencoders

An Online Projection Estimator for Nonparametric Regression in Reproducing Kernel Hilbert Spaces

The Kolmogorov Infinite Dimensional Equation in a Hilbert space Via Deep Learning Methods

How much is optimal reinsurance degraded by error?

Fast Learning Rate of Non-Sparse Multiple Kernel Learning and Optimal Regularization Strategies

Sub-linear convergence of a stochastic proximal iteration method in Hilbert space

This week in AI

1 Introduction

2 Integral representation of deep neural network

3 Finite approximation of the integral form

Assumption 1

Assumption 2

Assumption 3

Theorem 1 (Finite approximation error bound of the nonparametric model)

Lemma 1

4 Generalization error bounds

4.1 Notations

4.2 Empirical risk minimization

Theorem 2

4.3 Bayes estimator

Theorem 3

Corollary 1

4.4 Examples

4.4.1 Finite dimensional internal layer

4.4.2 Polynomial decreasing rate of eigenvalues