Is deeper better? It depends on locality of relevant features

1 Introduction

Deep learning has achieved an unparalleled success in various tasks of artificial intelligence such as image classification [1, 2] and speech recognition [3]. Remarkably, in modern machine learning applications, impressive generalization performance has been observed in an overparameterized regime, in which the number of parameters in the network is much larger than that of training data samples. Contrary to what we learn in the classical learning theory, an overparameterized network fits random labels and yet generalizes very well without serious overfitting [4]. We do not have general theory that explains why deep learning works so well.

Recently, the learning dynamics and the generalization power of heavily overparameterized wide neural networks have extensively been studied. It has been reported that training of an overparameterized network easily achieves zero training error without getting stuck in local minima of the loss landscape [4, 5]. Mathematically rigorous results have also been obtained [6, 7]. From a different point of view, theory of the neural tangent kernel (NTK) has been developed as a new tool to investigate an overparameterized network with an infinite width [8, 9], which simply explains the reason why a sufficiently wide neural network can achieve a global minimum of the training loss.

As for generalization, “double-descent” phenomenon has attracted much attention [10, 11]

. The standard bias-variance tradeoff picture predicts a U-shaped curve of the test error

[12]

, but instead we find the double-descent curve, which tells us that the increased model capacity beyond the interpolation threshold results in improved performance. This finding triggered detailed studies on the behavior of the bias and variance in an overparameterized regime

[13, 14]. The double-descent phenomenon is not explained by traditional complexity measures such as the Vapnik-Chervonenkis dimension and the Rademacher complexity [15], and hence one seeks for new complexity measures of deep neural networks that can prove better generalization bounds [16, 17, 18, 19, 20, 21].

These theoretical efforts mainly focus on the effect of increasing the network width, while benefits of the network depth remain unclear. It is known that expressivity of a deep neural network grows exponentially with the depth rather than the width [22]. See also [23, 24]. However, it is far from clear whether exponential expressivity really leads to better generalization [25, 26]. It is also nontrivial whether typical problems encountered in practice require such high expressivity. Although some works [27, 28] have shown that there exist simple and natural functions that are efficiently approximated by a network with two hidden layers but not by a network with one hidden layer, a recent work [29] has demonstrated that a deep network can only learn functions that are well approximated by a shallow network by using a gradient-based optimization algorithm, which indicates that benefits of depth are not due to high expressivity of deep networks. Some other recent works have reported no clear advantage of the depth in an overparameterized regime [30, 31].

To gain an insight into the advantage of the depth, in the present paper, we report our experimental study on the depth and width dependences of generalization in abstract but simple, well-controlled classification tasks with fully connected neural networks. We find that whether a deep network outperforms a shallow one depends on the property of relevant features for a given classification rule.

In this work, we introduce local labels and global labels

, both of which give simple mappings between inputs and output class labels. By “local”, we mean that the label is determined only by a few components of the input vector. On the other hand, a global label is given by a sum of local terms and determined by all components of the input. Our experiments show strong depth-dependences of the generalization error for those simple input-output mappings. In particular, we find that

deeper is better for local labels, while shallower is better for global labels. The implication of this result is that the depth is not always advantageous, but the locality of relevant features would give us a clue for understanding the advantage the depth brings about.

We also compare the generalization performance of a trained network of a finite width with that of the kernel method with the NTK. The latter corresponds to the infinite-width limit of a fully connected network with a proper initialization and an infinitesimal learning rate [8], which is referred to as the NTK limit. It is found that even if the width increases, in many cases the generalization error with an optimal learning rate does not converge to the NTK limit. In such a case, a finite-width network shows much better generalization compared with the kernel learning with the NTK. In the NTK limit, the network parameters stay close to their initial values during training, which is called the lazy learning [32], and hence the result mentioned above indicates the importance of the feature learning, in which network parameters change to learn relevant features.

2 Setting

We consider a classification task with a training dataset $D = {(x_{μ}, y_{μ}) : μ = 1, 2, \dots, N}$ , where $x_{μ} \in R^{d}$ is an input data and $y_{μ} \in {1, 2, \dots, K}$ is its label. In this work, we consider the binary classification, $K = 2$ , unless otherwise stated.

2.1 Dataset

Each input $x = (x^{(1)}, x^{(2)}, \dots, x^{(d)})^{T}$ is a $d$

-dimensional vector taken from i.i.d. Gaussian random variables of zero mean and unit variance, where

a^{T}

is the transpose of a vector

a

. For each input

x

, we assign a label

y

according to one of the following rules.

$k$ -local label

We randomly fix integers ${i_{1}, i_{2}, \dots, i_{k}}$ with $1 \leq i_{1} < i_{2} < \dots < i_{k} \leq d$ . In the “ $k$ -local” label, the relevant feature is given by the product of the $k$ components of an input $x$ , that is, the label $y$ is determined by

y = {\begin{matrix} 1 & if x^{(i_{1})} x^{(i_{2})} \dots x^{(i_{k})} \geq 0; 2 & otherwise . \end{matrix}

(1)

This label is said to be local in the sense that $y$ is completely determined by just the $k$ components of an input $x$ .¹¹1The locality here does not necessarily imply that $k$ points $i_{1}, i_{2}, \dots, i_{k}$ are spatially close to each other. Such a use of the terminology “ $k$ -local” has been found in the field of quantum computation [33]. For fully connected networks considered in this paper, without loss of generality, we can choose $i_{1} = 1$ , $i_{2} = 2$ ,… $i_{k} = k$ because of the permutation symmetry with respect to indices of input vectors.

$k$ -global label

We again fix $1 \leq i_{1} < i_{2} < \dots < i_{k} \leq d$ . Let us define

M = d \sum j = 1 x^{(j + i_{1})} x^{(j + i_{2})} \dots x^{(j + i_{k})},

(2)

where the convention $x^{(d + i)} = x^{(i)}$ is used. The $k$ -global label $y$ for $x$ is defined by

y = {\begin{matrix} 1 & if M \geq 0; 2 & otherwise . \end{matrix}

(3)

The relevant feature $M$ for this label is given by a uniform sum of the product of $k$ components of the input vector. Every component of $x$ contributes to this “ $k$ -global” label, in contrast to the $k$ -local label with $k < d$ .

2.2 Network architecture

In the present work, we consider fully connected feedforward neural networks with $L$ hidden layers of width $H$ . We call $L$ and $H$ the depth and the width of the network, respectively. The output of the network $f (x)$ for an input vector $x \in R^{d}$ is determined as follows:

⎧ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎩ \begin{matrix} f (x) = z^{(L + 1)} = w^{(L + 1)} z^{(L)} + b^{(L + 1)}; z^{(l)} = φ (w^{(l)} z^{(l - 1)} + b^{(l)}) for % l = 1, 2, \dots, L; z^{(0)} = x, \end{matrix}

(4)

where $φ (x) = max {x, 0}$

is the component-wise ReLu activation function,

z^{(l)}

is the output of the

l

th layer, and

w^{(l)} \in ⎧ ⎪ ⎨ ⎪ ⎩ \begin{matrix} R^{K \times H} % for l = L + 1; R^{H \times H} for l = 2, 3, \dots, L; R^{H \times d} for l = 1, \end{matrix} b^{(l)} \in {\begin{matrix} R^{K} for l = L + 1; R^{H} for l = 1, 2, \dots, L \end{matrix}

(5)

are the weights and the biases, respectively. Let us denote by $w$ the set of all the weights and biases in the network. We focus on an overparameterized regime, i.e., the number of network parameters (the number of components of $w$ ) exceeds $N$ , the number of training data points.

2.3 Supervised learning

The network parameters $w$

are adjusted to correctly classify the training data. It is done by minimizing the softmax cross-entropy loss

L (w)

given by

L (w) = \frac{1}{N} N \sum μ = 1 ℓ (f (x_{μ}), y_{μ}), ℓ (f (x), y) = - ln \frac{e^{f_{y} (x)}}{\sum_{i = 1}^{K} e^{f_{i} (x)}} = - f_{y} (x) + ln K \sum i = 1 e^{f_{i} (x)},

(6)

where the $i$ th component of $f (x)$ is denoted by $f_{i} (x)$

. The main results of our paper do not change for other standard loss functions such as the mean-squared error.

The training of the network is done by the stochastic gradient descent (SGD) with learning rate

η

and the mini-batch size

B

. That is, for each mini-batch

B \subset D

with

| B | = B

, the network parameter

w_{t}

at time

t

is updated as

w_{t + 1} = w_{t} - η \nabla_{w} L_{B} (w), L_{B} (w) = \frac{1}{B} \sum μ \in B ℓ (f (x_{μ}), y_{μ}) .

(7)

Throughout the paper, we fix $B = 50$ . Meanwhile, we optimize $η > 0$ before training (explain the detail later). Biases are initialized to be zero, and weights are initialized using the Glorot initialization [34].²²2We also tried the He initialization [35] and confirmed that results are similar to the ones obtained by the Glorot initialization, in particular when input vectors are normalized as $∥ x ∥ = 1$ .

The trained network classifies an input $x_{μ}$ to the class ${^y}_{μ}$ given by $^yμ=argmaxi∈{1,2,…,K}fi(xμ)$ . Let us then define the training error as

E_{t r a i n} = \frac{1}{N} N \sum μ = 1 (1 - δ_{y_{μ}, {^y}_{μ}}),

(8)

that is the miss-classification rate for the training data $D$ . We train our network until $E_{t r a i n} = 0$ is achieved, i.e., all the training data samples are correctly classified, which is possible in an overparameterized regime.

For a training dataset $D$ , we first perform the 10-fold cross validation to optimize the learning rate $η$ under the Bayesian optimization method [36], and then perform the training via the SGD by using the full training dataset. In the optimization of $η$ , we try to minimize the miss-classification ratio for the validation data.

The generalization performance of a trained network is measured by computing the test error. We prepare the test data $D_{t e s t} = {(x_{μ}^{'}, y_{μ}^{'}) : μ = 1, 2, \dots, N_{t e s t}}$ independently from the training data $D$ . The test error $E_{t e s t}$ is defined as the miss-classification ratio for $D_{t e s t}$ , i,.e.,

E_{t e s t} = \frac{1}{N_{t e s t}} N_{t e s t} \sum μ = 1 (1 - δ_{y_{μ}^{'}, {^y}_{μ}^{'}}),

(9)

where ${^y}_{μ}^{'} = {a r g m a x}_{i} f_{i} (x_{μ}^{'})$ is the prediction of our trained network. In our experiment discussed in Sec. 3, we fix $N_{t e s t} = 10^{5}$ .

2.4 Neural Tangent Kernel

Suppose a network of depth $L$ and width $H$ with the output $f (x, w) \in R^{K}$ . When the network is sufficiently wide and the learning rate is sufficiently small, the network parameters $w$ stay close to their randomly initialized values $w_{0}$ during training, and hence $f (x, w)$ is approximated by a linear function of $w - w_{0}$ : $f (x, w) = f (x, w_{0}) + \nabla_{w} f (x, w) |_{w = w_{0}} \cdot (w - w_{0})$ . As a result, the minimization of the mean-squared error $L_{M S E} = (1 / N) \sum_{μ = 1}^{N} [f (x_{μ}, w) - {\to y}_{μ}]^{2}$ , where $\to y \in R^{K}$ is the one-hot representation of the label $y$ , is equivalent to the kernel regression with the NTK $Θ_{i j}^{(L)} (x, x^{'})$ ( $i, j = 1, 2, \dots, K$ ) that is defined as

Θ_{i j}^{(L)} (x, x^{'}) = lim H \to \infty E_{w} [(\nabla_{w} f_{i} (x, w))^{T} (\nabla_{w} f_{j} (x, w))],

(10)

where $E_{w}$ denotes the average over random initializations of $w$ [8].

Let us consider a network whose biases ${b^{(l)}}$ and weights ${w^{(l)}}$ are randomly initialized as $b_{i}^{(l)} = β B_{i}^{(l)}$ with $B_{i}^{(l)} \sim N (0, 1)$ and $w_{i j}^{(l)} = \sqrt{2 / n_{l - 1}} W_{i j}^{(l)}$ with $W_{i j}^{(l)} \sim N (0, 1)$ for every $l$ respectively, where $n_{l}$

is the number of neurons in the

l

th layer, i.e.,

n_{0} = d

n_{1} = n_{2} = \dots = n_{L} = H

. The parameter

β

controls the impact of bias terms, and we set

β = 0.1

in our numerical experiment following Jacot et al. [8]. By using the ReLu activation function, we can give an explicit expression of the NTK that is suited for numerical calculations. Such formulas are given in Supplimentary Material.

It is shown that the NTK takes the form $Θ_{i j}^{(L)} (x, x^{'}) = δ_{i, j} Θ^{(L)} (x, x^{'})$ , and the minimization of the mean-squared error with an infinitesimal weight decay yields the output function

f^{N T K} (x) = N \sum μ, ν = 1 Θ^{(L)} (x, x_{μ}) {(K^{- 1})}_{μ ν}^{- 1} {\to y}_{ν},

(11)

where $K^{- 1}$ is the inverse matrix of the Gram matrix $K_{μ ν} = Θ^{(L)} (x_{μ}, x_{ν})$ . An input data $x$ is classified to $^y=argmaxi∈{1,2,…,K}fNTKi(x)$ .

3 Experimental results

We now present our experimental results. For each data point, the training dataset $D$ is fixed and we optimize the learning rate $η$ via the 10-fold cross validation with the Bayesian optimization method (we used the package provided in [37]). We used the optimized $η$

to train our network. At every 50 epochs we compute the training error

E_{t r a i n}

, and we stop the training if

E_{t r a i n} = 0

. For the fixed dataset

D

and the optimized learning rate

η

, the training is performed 10 times and calculate the average and the standard deviation of test errors

E_{t e s t}

3.1 1-local and 1-global labels

Figure 1: Test error against the number of training data samples $N$ for several network architectures specified by the depth and width for (a) the 1-local label and (b) the 1-global label. Test errors calculated by the NTK of depth 1 and 7 are also plotted. Error bars are smaller than the symbols.

In the 1-local and 1-global labels, the relevant feature is a linear function of the input vector. Therefore, in principle, even a linear network can correctly classify the data. Figure 1

shows the generalization errors in nonlinear networks of the varying depth and width as well as those in the linear perceptron (the network of zero depth). The input dimension is set to be

d = 1000

. We also plotted test errors calculated by the NTK, but we postpone the discussion about the NTK until Sec. 3.3.

Figure 1 shows that in both 1-local and 1-global labels, the test error decreases with the network width, and a shallower network ( $L = 1$ ) shows better generalization compared with a deeper one ( $L = 7$ ). The linear perceptron shows the best generalization performance, which is natural because it is the simplest network that is capable of learning the relevant feature associated with the 1-local or 1-global label. Remarkably, test errors of nonlinear networks ( $L = 1$ and $L = 7$ ) are not too large compared with those of the linear perceptron, although nonlinear networks are much more complex than the linear perceptron.

For a given network architecture, we do not see any important difference between the results for 1-local and 1-global labels, which would be explained by the fact that these labels are transformed to each other via the Fourier transformation of input vectors.

3.2 Opposite depth dependences for $k$ -local and $k$ -global labels with $k \geq 2$

For $k \geq 2$ , it turns out that experimental results show opposite depth dependences for $k$ -local and $k$ -global labels. Let us first consider $k$ -local labels with $k \geq 2$ . Figure 2 (a) and (b) show test errors for varying $N$ in various networks for the 2-local and the 3-local labels, respectively. The input dimension $d$ is set to be $d = 500$ in the 2-local label and $d = 100$ in the 3-local label. We see that the test error strongly depends on the network depth. A deeper network ( $L = 7$ ) generalizes better than a shallower one ( $L = 1$ ). It should be noted that for $d = 500$ , the network of $L = 1$ and $H = 2000$ contains about $10^{6}$ trainable parameters, the number of which is much larger than that of trainable parameters ( $≃ 10^{5}$ ) in the network of $L = 7$ and $H = 100$ . In spite of this fact, the latter outperforms the former in the 2-local label as well as in the 3-local label with large $N$ , which implies that increasing the number of trainable parameters do not necessarily implies better generalization. In $k$ -local labels with $k \geq 2$ , the network depth is more strongly correlated to generalization compared with the network width.

From Fig. 2 (b), it is obvious that the network of $L = 7$ and $H = 100$ fails to learn the 3-local label for small $N$ . We also see that error bars of the test error are large in the network of $L = 7$ and $H = 100$ . The error bar represents the variance due to initialization and training. By increasing the network width $H$ , both variances and test errors decrease. This result is consistent with the recent observation in the lazy regime that increasing the network width results in better generalization because it reduces the variance due to initialization [14].

Figure 2: Test error against the number of training data samples $N$ for several network architectures specified by the depth and the width for (a) the 2-local label, (b) the 3-local label, (c) 2-global label, and (d) 3-global label. Error bars indicate the standard deviation of the test error for 10 iterations of the network initialization and the training. Test errors calculated by the NTK of the depth of 1 and 7 are also plotted.

Figure 3: Depth dependence of the test error for $N = 10^{4}$ training samples with 2-local and 2-global labels. The dimension of input vectors is set to be $d = 500$ in the 2-local label and $d = 100$ in the 2-global label. The network width is fixed to be 500. An error bar indicates the standard deviation over 10 iterations of the training using the same dataset.

Next, we consider $k$ -global labels with $k = 2$ and 3. The input dimension $d$ is set as $d = 100$ for the 2-global label and $d = 40$ for the 3-global label. We plot test errors against $N$ in Fig. 2 for (c) the 2-global label and (d) the 3-global label. Again we find strong depth dependences, but now shallow networks ( $L = 1$ ) outperform deep ones ( $L = 7$ ), which is contrary to the results for $k$ -local labels. For $L = 7$ , we also find strong width dependences; the test error of a wider network more quickly decreases with $N$ . In particular, in the 3-global label, an improvement of the generalization with $N$ is subtle for $L = 7$ and $H = 100$ . By increasing the width, the decrease of the test error with $N$ becomes much faster [see the result for $L = 7$ and $H = 500$ in Fig. 2 (d)].

To see more details of the effect of depth, we also plot the depth dependence of the test error for fixed training data samples. We prepare $N = 10000$ training data samples for the 2-local and 2-global labels, respectively. The input dimension is $d = 500$ for the 2-local label and $d = 100$ for the 2-global label. By using the prepared training data samples, networks of the depth $L$ and the width $H = 500$ are trained up to $L = 10$ . The test errors of trained networks are shown in Fig. 3. In the 2-local label, the test error decreases with $L$ , whereas the test error increases with $L$ in the 2-global label. Thus, Fig. 3 clearly shows the opposite depth dependences for local and global labels.

3.3 Comparison between finite networks and NTKs

In Figs. 1 and 2, test errors calculated by using the NTK are also plotted. In the case of $k = 1$ (Fig. 1), the generalization performance of the NTK is comparable with that of finite networks. For the 2-global label [Fig. 2 (c)], the test error obtained by the NTK is comparable or lower than that of finite networks.

The crucial difference is seen in the case of $k$ -local label with $k = 2$ and 3 and the 3-global label. In Fig. 2 (a) and (b), we see that the NTK almost completely fails to classify the data, although finite networks succeed in doing so. In the case of the 3-global label, the NTK of depth $L = 7$ correctly classifies the data, while the NTK of depth $L = 1$ fails [see Fig. 2 (d)]. In those cases, the test error calculated by a finite network does not seem to converge to that obtained by the NTK as the network width increases.

The NTK has been proposed as a theoretical tool to investigate the infinite-width limit, but it should be kept in mind that the learning rate has to be sufficiently small to achieve the NTK limit [8, 9]. The discrepancy between a wide network and the NTK in Fig. 2 stems from the strong learning-rate dependence of the generalization performance. In our experiment, the learning rate has been optimized by performing the 10-fold cross validation. If the optimized learning rate is not small enough for each width, the trained network may not be described by the NTK even in the infinite-width limit.

In Fig. 4 (a), we plot the learning-rate dependence of the test error for the 2-local label and the 2-global label in the network of the depth $L = 1$ and the width $H = 2000$ . We observe a sharp learning-rate dependence in the case of the 2-local label in contrast to the case of the 2-global label. In Fig. 4 (b), we compare the learning-rate dependences of the test error for $L = 1$ and $L = 7$ in the case of the 3-global label (in both cases $H = 2000$ ). We see that the learning-rate dependence for $L = 1$ is much stronger than that for $L = 7$ , which is consistent with the fact that the NTK fails only for $L = 1$ . It should be noted that Fig. 4 (b) shows that the deep network ( $L = 7$ ) outperforms the shallow one ( $L = 1$ ) in the regime of small learning rates, while the shallow one performs better than the deep one at their optimal learning rates.

Figure 4 also shows that the test error for a sufficiently small learning rate approaches the one obtained by the corresponding NTK. Therefore, the regime of small learning rates is identified as a lazy learning regime, while larger learning rates correspond to a feature learning regime. Sharp learning-rate dependences found here provide theoretical and practical importance of the feature learning.

Figure 4: Learning-rate dependence of the test error. (a) Numerical results for the 2-local and 2-global labels in the network with the depth of 1 and the width of 2000. (b) Numerical results for the 3-global label in the networks with the depth of 1 and 7 (the network width is set at 2000 for both cases). The dotted lines show the test error calculated by the NTK. When the learning rate is sufficiently small, the test error in a finite network approaches that of the corresponding NTK. Each data is plotted up to the maximum learning rate beyond which the zero training error is not achieved within 2500 epochs (in some cases training fails due to divergence of network parameters during the training). Error bars indicate the standard deviation over 10 iterations of the training.

4 Conclusion

In this work, we have studied the effect of increasing the depth in classification tasks. Instead of using real data, we have employed abstract setting with random inputs and simple classification rules because such a simple setup helps us understand under what situations deeper networks perform better or worse. We find that the locality of the relevant feature for a given classification rule plays a key role.

We note that the advantage of the depth in local labels is not due to high expressivity of deep networks. If a network can accurately classify the data with the $k$ -local label and the input dimension $d = k$ , it can in principle classify the data with an arbitrarily large input dimension $d \geq k$ . This is because the $k$ -local label depends only on the $k$ components among $d$ components. By using this fact, it is confirmed that a small network with one hidden layer of the width of about 10-100 can express the 2-local label and the 3-local label almost perfectly.³³3This fact does not mean at all that such a small network can actually learn the local label for large $d$ by a gradient-based algorithm. In other words, learning the $k$ -local label for small $k$ does not require high expressive power. Nevertheless, a deeper network outperforms a shallower one.

It is also an interesting observation that shallower networks do better than deeper ones for the $k$ -global label. This result shows that the depth is not always beneficial. In future studies, we hope to investigate which properties of the data other than locality studied here result in (dis)advantage of the depth.

Broader Impact

It is an important practical problem to find neural architecture designing principles depending on specific machine-learning tasks. Although our paper is motivated by a theoretical question, i.e., why deep networks perform so well compared with shallow networks, this work will impact the practical problem mentioned above. Our work indicates that a deeper architecture is better for local features, whereas a shallower architecture is better for global features. It is expected that local features are important in typical image classification tasks, and our work suggests that a deep architecture should be used for such a task, which is consistent with our experience. Furthermore, if we could find a transformation of local features to global ones, then even shallow networks should be able to classify the data with a great accuracy after the transformation, which is of practical merit. It would be also important to identify problems where labels are effectively global (e.g., where Fourier analysis works well, finance, weather forecasting). The present research can lead to better solutions in these fields. In short, our theoretical work suggests a guiding principle for future studies on architecture designing principles.

References

Krizhevsky et al. [2012]
A. Krizhevsky, I. Sutskever, and G. E. Hinton, Imagenet classification with deep convolutional neural networks, in
Advances in Neural Information Processing Systems (2012) pp. 1097–1105.
LeCun et al. [2015] Y. LeCun, Y. Bengio, and G. Hinton, Deep learning, Nature 521, 436–444 (2015).
Hinton et al. [2012] G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, and Others, Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups, IEEE Signal processing magazine 29, 82–97 (2012).
Zhang et al. [2017] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, Understanding Deep Learning Requires Rethinking of Generalization, in International Conference on Learning Representations (2017).
Baity-Jesi et al. [2018] M. Baity-Jesi, L. Sagun, M. Geiger, S. Spigler, G. Ben Arous, C. Cammarota, Y. LeCun, M. Wyart, and G. Biroli, Comparing Dynamics : Deep Neural Networks versus Glassy Systems, in International Conference on Machine Learning (2018).
Allen-Zhu et al. [2019] Z. Allen-Zhu, Y. Li, and Z. Song, A convergence theory for deep learning via over-parameterization, in International Conference on Machine Learning (2019) arXiv:1811.03962.
Du et al. [2019] S. S. Du, J. D. Lee, H. Li, L. Wang, and X. Zhai, Gradient descent finds global minima of deep neural networks, in International Conference on Machine Learning (2019) arXiv:1811.03804.
Jacot et al. [2018] A. Jacot, F. Gabriel, and C. Hongler, Neural tangent kernel: Convergence and generalization in neural networks, in Advances in Neural Information Processing Systems (2018) arXiv:1806.07572.
Arora et al. [2019] S. Arora, S. S. Du, W. Hu, Z. Li, R. Salakhutdinov, and R. Wang, On Exact Computation with an Infinitely Wide Neural Net, in Neural Information Processing Systems (2019) arXiv:1904.11955.
Spigler et al. [2019] S. Spigler, M. Geiger, S. D’Ascoli, L. Sagun, G. Biroli, and M. Wyart, A jamming transition from under- To over-parametrization affects generalization in deep learning, Journal of Physics A: Mathematical and Theoretical 52, 474001 (2019), arXiv:1810.09665.
Belkin et al. [2019] M. Belkin, D. Hsu, S. Ma, and S. Mandal, Reconciling modern machine-learning practice and the classical biasâvariance trade-off, Proceedings of the National Academy of Sciences of the United States of America 116, 15849–15854 (2019), arXiv:1812.11118.
Geman et al. [1992] S. Geman, E. Bienenstock, and R. Doursat, Neural Networks and the Bias/Variance Dilemma, Neural Computation 4, 1–58 (1992).
Neal et al. [2019] B. Neal, S. Mittal, A. Baratin, V. Tantia, M. Scicluna, S. Lacoste-Julien, and I. Mitliagkas, A Modern Take on the Bias-Variance Tradeoff in Neural Networks, in Workshop on Identifying and Understanding Deep Learning Phenomena (2019) arXiv:1810.08591.
[14] S. D’Ascoli, M. Refinetti, G. Biroli, and F. Krzakala, Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime, arXiv:2003.01054.
Mohri et al. [2018] M. Mohri, A. Rostamizadeh, and A. Talwalkar, Foundations of Machine Learning (MIT press, 2018).
Dziugaite and Roy [2017] G. K. Dziugaite and D. M. Roy, Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data, in Uncertainty in Artificial Intelligence (2017) arXiv:1703.11008.
Neyshabur et al. [2017] B. Neyshabur, S. Bhojanapalli, D. Mcallester, and N. Srebro, Exploring Generalization in Deep Learning, in Advances in Neural Information Processing Systems (2017).
Neyshabur et al. [2019] B. Neyshabur, Z. Li, S. Bhojanapalli, Y. LeCun, and N. Srebro, The role of over-parametrization in generalization of neural networks, in International Conference on Learning Representations (2019) arXiv:1805.12076.
Arora et al. [2018] S. Arora, R. Ge, B. Neyshabur, and Y. Zhang, Stronger generalization bounds for deep nets via a compression approach, in International Conference on Machine Learning (2018) arXiv:1802.05296 .
Nagarajan and Kolter [2017] V. Nagarajan and J. Z. Kolter, Generalization in Deep Networks: The Role of Distance from Initialization, in Advances in Neural Information Processing Systems (2017) arXiv:1901.01672.
Pérez et al. [2019] G. V. Pérez, A. A. Louis, and C. Q. Camargo, Deep learning generalizes because the parameter-function map is biased towards simple functions, in International Conference on Learning Representations (2019) arXiv:1805.08522.
Poole et al. [2016] B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein, and S. Ganguli, Exponential expressivity in deep neural networks through transient chaos, in Advances in Neural Information Processing Systems (2016) arXiv:1606.05340.
Bianchini and Scarselli [2014] M. Bianchini and F. Scarselli, On the complexity of neural network classifiers: A comparison between shallow and deep architectures, IEEE Transactions on Neural Networks and Learning Systems 25, 1553–1565 (2014).
Montúfar et al. [2014] G. Montúfar, R. Pascanu, K. Cho, and Y. Bengio, On the number of linear regions of deep neural networks, in Advances in Neural Information Processing Systems (2014) arXiv:1402.1869.
Ba and Caruana [2014] J. Ba and R. Caruana, Do deep networks really need to be deep?, in Advances in Neural Information Processing Systems (2014) arXiv:1312.6184.
Becker et al. [2020] S. Becker, Y. Zhang, and A. A. Lee, Geometry of Energy Landscapes and the Optimizability of Deep Neural Networks, Physical Review Letters 124, 108301 (2020), arXiv:1808.00408.
Eldan and Shamir [2016] R. Eldan and O. Shamir, The Power of Depth for Feedforward Neural Networks, in Proceedings of Machine Learning Research (2016).
Safran and Shamir [2017] I. Safran and O. Shamir, Depth-width tradeoffs in approximating natural functions with neural networks, in International Conference on Machine Learning (2017) arXiv:1610.09887.
Malach and Shalev-Shwartz [2019] E. Malach and S. Shalev-Shwartz, Is Deeper Better only when Shallow is Good?, in Advances in Neural Information Processing Systems (2019) arXiv:1903.03488.
Geiger et al. [2019] M. Geiger, S. Spigler, S. D’Ascoli, L. Sagun, M. Baity-Jesi, G. Biroli, and M. Wyart, Jamming transition as a paradigm to understand the loss landscape of deep neural networks, Physical Review E 100, 012115 (2019), arXiv:1809.09349.
[31] M. Geiger, S. Spigler, A. Jacot, and M. Wyart, Disentangling feature and lazy training in deep neural networks, arXiv:1906.08034.
Chizat et al. [2019] L. Chizat, E. Oyallon, and F. Bach, On Lazy Training in Differentiable Programming, in Neural Information Processing Systems (2019) arXiv:1812.07956.
Kempe et al. [2006] J. Kempe, A. Kitaev, and O. Regev, The complexity of the local Hamiltonian problem, SIAM Journal on Computing 35, 1070–1097 (2006).
Glorot and Bengio [2010] X. Glorot and Y. Bengio, Understanding the difficulty of training deep feedforward neural networks, in Proceedings of Machine Learning Research (2010).
He et al. [2015] K. He, X. Zhang, S. Ren, and J. Sun, Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, in Proceedings of the IEEE International Conference on Computer Vision (2015) arXiv:1502.01852.
Snoek et al. [2012] J. Snoek, H. Larochelle, and R. P. Adams, Practical Bayesian Optimization of Machine Learning Algorithms, in Advances in Neural Information Processing Systems (2012).
Nogueira [2014] F. Nogueira, Bayesian Optimization: Open source constrained global optimization tool for Python (2014).

Appendix A Explicit expression of the NTK

We consider a network whose biases ${b^{(l)}}$ and weights ${w^{(l)}}$ are randomly initialized as $b_{i}^{(l)} = β B_{i}^{(l)}$ with $B_{i}^{(l)} \sim N (0, 1)$ and $w_{i j}^{(l)} = \sqrt{2 / n_{l - 1}} W_{i j}^{(l)}$ with $W_{i j}^{(l)} \sim N (0, 1)$ for every $l$ , where $n_{l}$ is the number of neurons in the $l$ th layer, i.e., $n_{0} = d$ , $n_{1} = n_{2} = \dots = n_{L} = H$ . In the infinite-width limit $H \to \infty$ , the pre-activation $f^{(l)} = w^{(l)} z^{(l - 1)} + b^{(l)}$ at every hidden layer tends to an i.i.d. Gaussian process with covariance $Σ^{(l - 1)} : R^{d} \times R^{d} \to R$ that is defined recursively as

(12)

for $l = 1, 2, \dots, L$ . We also define

{˙ Σ}^{(l)} (x, x^{'}) = 2 E_{(u, v) \sim N (0, Λ^{(l)})} [˙ φ (u) ˙ φ (v)],

(13)

where $˙ φ$ is the derivative of $φ$ . The NTK is then expressed as $Θ_{i j}^{(L)} (x, x^{'}) = δ_{i, j} Θ^{(L)} (x, x^{'})$ , where

Θ^{(L)} (x, x^{'}) = L + 1 \sum l = 1 (Σ^{(l - 1)} (x, x^{'}) L + 1 \prod l^{'} = l {˙ Σ}^{(l^{'})} (x, x^{'})) .

(14)

The derivation of this formula is given by Arora et al. [9].

Using the ReLu activation function $φ (u) = max {u, 0}$ , we can further calculate $Σ^{(l)} (x, x^{'})$ and ${˙ Σ}^{(l)} (x, x^{'})$ , obtaining

Σ^{(l)} (x, x^{'}) = \frac{\sqrt{det Λ^{(l)}}}{π} + \frac{Σ^{(l - 1)} (x, x^{'})}{π} [\frac{π}{2} + arctan (\frac{Σ^{(l - 1)} (x, x^{'})}{\sqrt{det Λ^{(l)}}})] + β^{2}

(15)

and

{˙ Σ}^{(l)} (x, x^{'}) = \frac{1}{2} [1 + \frac{2}{π} arctan (\frac{Σ^{(l - 1)} (x, x^{'})}{\sqrt{det Λ^{(l)}}})] .

(16)

For $x = x^{'}$ , we obtain $Σ^{(l)} (x, x) = Σ^{(0)} (x, x) + l β^{2} = ∥ x ∥^{2} / d + (l + 1) β^{2}$ . By solving eqs. (15) and (16) iteratively, the NTK in eq. (14) is obtained.⁴⁴4When $β = 0$ (no bias), the equations are further simplified; $Σ^{(l)} = \frac{∥ x ∥ ∥ x^{'} ∥}{d} cos θ^{(l)}$ and ${˙ Σ}^{(l)} = 1 - \frac{θ^{(l - 1)}}{π}$ , where $θ^{(0)} \in [0, π]$ is the angle between $x$ and $x^{'}$ , and $θ^{(l)}$ is iteratively determined by

cos θ^{(l)} = \frac{1}{π} [sin θ^{(l - 1)} + (π - θ^{(l - 1)}) cos θ^{(l - 1)}] .

Is deeper better? It depends on locality of relevant features

Interplay between depth of neural networks and locality of target functions

Finite Depth and Width Corrections to the Neural Tangent Kernel

Robustness in deep learning: The good (width), the bad (depth), and the ugly (initialization)

Effects of Depth, Width, and Initialization: A Convergence Analysis of Layer-wise Training for Deep Linear Neural Networks

Is Feature Diversity Necessary in Neural Network Initialization?

Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks

Deep Residual Local Feature Learning for Speech Emotion Recognition

1 Introduction